[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786512017613366-32144-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 05:20:17 dist-test-slave-42z9 krb5kdc[32153](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 05:20:17 dist-test-slave-42z9 krb5kdc[32153](info): set up 2 sockets
Aug 12 05:20:17 dist-test-slave-42z9 krb5kdc[32153](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260812 05:20:19.029422 32144 mini_kdc.cc:121] Time spent starting KDC: real 1.398s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786512017613366-32144-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786512017613366-32144-0/krb5kdc/test-user.keytab.
Aug 12 05:20:19 dist-test-slave-42z9 krb5kdc[32153](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512019, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T05:20:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T05:20:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.100.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.100.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.100.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.62.keytab.
Entry for principal kudu/127.31.100.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.62.keytab.
I20260812 05:20:19.145572 32144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0waFvT/build/release/bin/kudu
/tmp/dist-test-task0waFvT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786512017613366-32144-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786512017613366-32144-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786512017613366-32144-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786512017613366-32144-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.100.62:42079
--webserver_interface=127.31.100.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.62.keytab
--principal=kudu/127.31.100.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.100.20:36793
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.100.62:42079
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786512017613366-32144-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786512017613366-32144-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786512017613366-32144-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 05:20:19.229597 32175 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 05:20:19.229787 32175 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 05:20:19.229808 32175 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 05:20:19.230964 32175 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 05:20:19.231003 32175 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 05:20:19.231016 32175 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 05:20:19.231029 32175 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 05:20:19.231053 32175 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 05:20:19.232520 32175 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.100.20:36793
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786512017613366-32144-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786512017613366-32144-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.100.62:42079
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.62.keytab
--principal=kudu/127.31.100.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.100.62:42079
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786512017613366-32144-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.100.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32175
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786512017613366-32144-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 04:20:34 UTC on ccycloud.quasar-phqdqp.root.comops.site
build id 4581871
I20260812 05:20:19.232735 32175 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 05:20:19.232928 32175 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 05:20:19.237715 32180 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 05:20:19.237766 32175 server_base.cc:1034] running on GCE node
W20260812 05:20:19.237722 32186 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 05:20:19.237732 32181 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 05:20:19.238111 32175 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 05:20:19.238386 32175 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 05:20:19.239537 32175 hybrid_clock.cc:648] HybridClock initialized: now 1786512019239522 us; error 29 us; skew 500 ppm
Aug 12 05:20:19 dist-test-slave-42z9 krb5kdc[32153](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512019, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.100.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 05:20:19.241905 32175 init.cc:377] Logged in from keytab as kudu/127.31.100.62@KRBTEST.COM (short username kudu)
I20260812 05:20:19.242331 32175 webserver.cc:460] Webserver started at http://127.31.100.62:33015/ using document root <none> and password file <none>
I20260812 05:20:19.242538 32175 fs_manager.cc:359] Metadata directory not provided
I20260812 05:20:19.242587 32175 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 05:20:19.242724 32175 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 05:20:19.243795 32175 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786512017613366-32144-0/minicluster-data/master-0/data/instance:
uuid: "8450325ba1574e759059ca19bca25993"
format_stamp: "Formatted at 2026-08-12 05:20:19 on dist-test-slave-42z9"
server_key: "51fc3e5174d7ec11951ee109925c6fee"
server_key_iv: "e8ad17cfbe146323dd394ba65463b405"
server_key_version: "encryptionkey@0"
I20260812 05:20:19.244120 32175 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786512017613366-32144-0/minicluster-data/master-0/wal/instance:
uuid: "8450325ba1574e759059ca19bca25993"
format_stamp: "Formatted at 2026-08-12 05:20:19 on dist-test-slave-42z9"
server_key: "51fc3e5174d7ec11951ee109925c6fee"
server_key_iv: "e8ad17cfbe146323dd394ba65463b405"
server_key_version: "encryptionkey@0"
I20260812 05:20:19.245381 32175 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260812 05:20:19.246088 32197 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 05:20:19.246234 32175 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260812 05:20:19.246317 32175 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786512017613366-32144-0/minicluster-data/master-0/data,/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786512017613366-32144-0/minicluster-data/master-0/wal
uuid: "8450325ba1574e759059ca19bca25993"
format_stamp: "Formatted at 2026-08-12 05:20:19 on dist-test-slave-42z9"
server_key: "51fc3e5174d7ec11951ee109925c6fee"
server_key_iv: "e8ad17cfbe146323dd394ba65463b405"
server_key_version: "encryptionkey@0"
I20260812 05:20:19.246376 32175 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786512017613366-32144-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786512017613366-32144-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786512017613366-32144-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 05:20:19.254308 32175 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 05:20:19.256260 32175 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 05:20:19.256385 32175 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 05:20:19.264580 32175 rpc_server.cc:307] RPC server started. Bound to: 127.31.100.62:42079
I20260812 05:20:19.264634 32286 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.100.62:42079 every 8 connection(s)
I20260812 05:20:19.265007 32175 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786512017613366-32144-0/minicluster-data/master-0/data/info.pb
I20260812 05:20:19.265961 32287 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 05:20:19.268877 32287 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8450325ba1574e759059ca19bca25993: Bootstrap starting.
I20260812 05:20:19.270000 32144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0waFvT/build/release/bin/kudu as pid 32175
I20260812 05:20:19.270072 32287 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8450325ba1574e759059ca19bca25993: Neither blocks nor log segments found. Creating new log.
I20260812 05:20:19.270092 32144 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786512017613366-32144-0/minicluster-data/master-0/wal/instance
I20260812 05:20:19.270808 32144 external_mini_cluster.cc:1428] Setting key 7bd6147b5efdc63bbf34cb23b87645c4
I20260812 05:20:19.271049 32287 log.cc:824] T 00000000000000000000000000000000 P 8450325ba1574e759059ca19bca25993: Log is configured to *not* fsync() on all Append() calls
I20260812 05:20:19.272074 32287 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8450325ba1574e759059ca19bca25993: No bootstrap required, opened a new log
I20260812 05:20:19.276507 32287 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8450325ba1574e759059ca19bca25993 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8450325ba1574e759059ca19bca25993" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 42079 } }
I20260812 05:20:19.276654 32287 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8450325ba1574e759059ca19bca25993 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 05:20:19.276687 32287 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8450325ba1574e759059ca19bca25993 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8450325ba1574e759059ca19bca25993, State: Initialized, Role: FOLLOWER
I20260812 05:20:19.276835 32287 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8450325ba1574e759059ca19bca25993 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8450325ba1574e759059ca19bca25993" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 42079 } }
I20260812 05:20:19.276935 32287 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8450325ba1574e759059ca19bca25993 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 05:20:19.276978 32287 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8450325ba1574e759059ca19bca25993 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 05:20:19.277031 32287 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8450325ba1574e759059ca19bca25993 [term 0 FOLLOWER]: Advancing to term 1
I20260812 05:20:19.278494 32287 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8450325ba1574e759059ca19bca25993 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8450325ba1574e759059ca19bca25993" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 42079 } }
I20260812 05:20:19.278611 32287 leader_election.cc:302] T 00000000000000000000000000000000 P 8450325ba1574e759059ca19bca25993 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8450325ba1574e759059ca19bca25993; no voters:
I20260812 05:20:19.278784 32287 leader_election.cc:288] T 00000000000000000000000000000000 P 8450325ba1574e759059ca19bca25993 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 05:20:19.278815 32297 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8450325ba1574e759059ca19bca25993 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 05:20:19.278951 32297 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8450325ba1574e759059ca19bca25993 [term 1 LEADER]: Becoming Leader. State: Replica: 8450325ba1574e759059ca19bca25993, State: Running, Role: LEADER
I20260812 05:20:19.279064 32287 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8450325ba1574e759059ca19bca25993 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 05:20:19.279058 32297 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8450325ba1574e759059ca19bca25993 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8450325ba1574e759059ca19bca25993" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 42079 } }
I20260812 05:20:19.279795 32299 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8450325ba1574e759059ca19bca25993 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8450325ba1574e759059ca19bca25993" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8450325ba1574e759059ca19bca25993" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 42079 } } }
I20260812 05:20:19.279909 32299 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8450325ba1574e759059ca19bca25993 [sys.catalog]: This master's current role is: LEADER
I20260812 05:20:19.280093 32301 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8450325ba1574e759059ca19bca25993 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8450325ba1574e759059ca19bca25993. Latest consensus state: current_term: 1 leader_uuid: "8450325ba1574e759059ca19bca25993" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8450325ba1574e759059ca19bca25993" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 42079 } } }
I20260812 05:20:19.280174 32301 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8450325ba1574e759059ca19bca25993 [sys.catalog]: This master's current role is: LEADER
I20260812 05:20:19.280642 32306 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 05:20:19.281102 32306 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 05:20:19.283530 32306 catalog_manager.cc:1347] Generated new cluster ID: a897a029e60543a887db3285505bddbe
I20260812 05:20:19.283576 32306 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 12 05:20:19 dist-test-slave-42z9 krb5kdc[32153](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512019, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.100.62@KRBTEST.COM
I20260812 05:20:19.292711 32291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:19.271753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48614 (local address 127.31.100.62:42079)
0812 05:20:19.272012 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0812 05:20:19.272022 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 05:20:19.272449 (+ 427us) server_negotiation.cc:407] Connection header received
0812 05:20:19.276535 (+ 4086us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:20:19.276540 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:20:19.276582 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:20:19.276691 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:20:19.279446 (+ 2755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:19.282177 (+ 2731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:19.283692 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:19.284266 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:19.284385 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:19.288739 (+ 4354us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 05:20:19.288744 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 05:20:19.288745 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 05:20:19.288758 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0812 05:20:19.290761 (+ 2003us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:20:19.291232 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:20:19.291235 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:20:19.291235 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:20:19.291369 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:20:19.291615 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:20:19.291616 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:20:19.291616 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:20:19.291920 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 05:20:19.292079 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0812 05:20:19.292330 (+ 251us) server_negotiation.cc:299] Negotiation successful
0812 05:20:19.292483 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":197,"thread_start_us":146,"threads_started":1}
I20260812 05:20:19.303557 32306 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 05:20:19.304252 32306 catalog_manager.cc:1504] Loading token signing keys...
I20260812 05:20:19.312791 32306 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8450325ba1574e759059ca19bca25993: Generated new TSK 0
I20260812 05:20:19.312948 32306 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.31.100.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.100.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.100.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.1.keytab.
Entry for principal kudu/127.31.100.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.1.keytab.
I20260812 05:20:19.344009 32144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0waFvT/build/release/bin/kudu
/tmp/dist-test-task0waFvT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786512017613366-32144-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786512017613366-32144-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786512017613366-32144-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786512017613366-32144-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.100.1:0
--local_ip_for_outbound_sockets=127.31.100.1
--webserver_interface=127.31.100.1
--webserver_port=0
--tserver_master_addrs=127.31.100.62:42079
--keytab_file=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.1.keytab
--principal=kudu/127.31.100.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.100.20:36793
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786512017613366-32144-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786512017613366-32144-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786512017613366-32144-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 05:20:19.432204 32333 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 05:20:19.432392 32333 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 05:20:19.432415 32333 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 05:20:19.433557 32333 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 05:20:19.433597 32333 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 05:20:19.433625 32333 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.100.1
I20260812 05:20:19.435063 32333 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.100.20:36793
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786512017613366-32144-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786512017613366-32144-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.1.keytab
--principal=kudu/127.31.100.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.100.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786512017613366-32144-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.100.1
--webserver_port=0
--tserver_master_addrs=127.31.100.62:42079
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32333
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.100.1
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786512017613366-32144-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 04:20:34 UTC on ccycloud.quasar-phqdqp.root.comops.site
build id 4581871
I20260812 05:20:19.435271 32333 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 05:20:19.435488 32333 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 05:20:19.440300 32343 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 05:20:19.440305 32345 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 05:20:19.440306 32342 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 05:20:19.440577 32333 server_base.cc:1034] running on GCE node
I20260812 05:20:19.440737 32333 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 05:20:19.440927 32333 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 05:20:19.442065 32333 hybrid_clock.cc:648] HybridClock initialized: now 1786512019442051 us; error 32 us; skew 500 ppm
Aug 12 05:20:19 dist-test-slave-42z9 krb5kdc[32153](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512019, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.100.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 05:20:19.444458 32333 init.cc:377] Logged in from keytab as kudu/127.31.100.1@KRBTEST.COM (short username kudu)
I20260812 05:20:19.444847 32333 webserver.cc:460] Webserver started at http://127.31.100.1:45813/ using document root <none> and password file <none>
I20260812 05:20:19.445017 32333 fs_manager.cc:359] Metadata directory not provided
I20260812 05:20:19.445055 32333 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 05:20:19.445142 32333 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 05:20:19.446100 32333 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786512017613366-32144-0/minicluster-data/ts-0/data/instance:
uuid: "613d489725344176868fd3db02b78b67"
format_stamp: "Formatted at 2026-08-12 05:20:19 on dist-test-slave-42z9"
server_key: "9eedefaa06b857efb5a5ceaa22fd25ac"
server_key_iv: "c8eaca69b3220043938004af64032b61"
server_key_version: "encryptionkey@0"
I20260812 05:20:19.446377 32333 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786512017613366-32144-0/minicluster-data/ts-0/wal/instance:
uuid: "613d489725344176868fd3db02b78b67"
format_stamp: "Formatted at 2026-08-12 05:20:19 on dist-test-slave-42z9"
server_key: "9eedefaa06b857efb5a5ceaa22fd25ac"
server_key_iv: "c8eaca69b3220043938004af64032b61"
server_key_version: "encryptionkey@0"
I20260812 05:20:19.447546 32333 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260812 05:20:19.448189 32353 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 05:20:19.448329 32333 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260812 05:20:19.448410 32333 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786512017613366-32144-0/minicluster-data/ts-0/data,/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786512017613366-32144-0/minicluster-data/ts-0/wal
uuid: "613d489725344176868fd3db02b78b67"
format_stamp: "Formatted at 2026-08-12 05:20:19 on dist-test-slave-42z9"
server_key: "9eedefaa06b857efb5a5ceaa22fd25ac"
server_key_iv: "c8eaca69b3220043938004af64032b61"
server_key_version: "encryptionkey@0"
I20260812 05:20:19.448482 32333 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786512017613366-32144-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786512017613366-32144-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786512017613366-32144-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 05:20:19.460539 32333 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 05:20:19.462484 32333 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 05:20:19.462604 32333 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 05:20:19.462798 32333 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 05:20:19.463104 32333 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 05:20:19.463140 32333 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 05:20:19.463160 32333 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 05:20:19.463182 32333 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 05:20:19.469851 32333 rpc_server.cc:307] RPC server started. Bound to: 127.31.100.1:36323
I20260812 05:20:19.469892 32545 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.100.1:36323 every 8 connection(s)
I20260812 05:20:19.470165 32333 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786512017613366-32144-0/minicluster-data/ts-0/data/info.pb
Aug 12 05:20:19 dist-test-slave-42z9 krb5kdc[32153](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512019, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.100.1@KRBTEST.COM for kudu/127.31.100.62@KRBTEST.COM
I20260812 05:20:19.478605 32144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0waFvT/build/release/bin/kudu as pid 32333
I20260812 05:20:19.478695 32144 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786512017613366-32144-0/minicluster-data/ts-0/wal/instance
I20260812 05:20:19.478854 32144 external_mini_cluster.cc:1428] Setting key b4c7c5802c927dc59f8fe48008d70f86
I20260812 05:20:19.481666 32291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:19.470813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.100.1:49099 (local address 127.31.100.62:42079)
0812 05:20:19.470889 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0812 05:20:19.470891 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 05:20:19.471055 (+ 164us) server_negotiation.cc:407] Connection header received
0812 05:20:19.471482 (+ 427us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:20:19.471483 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:20:19.471497 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:20:19.471543 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:20:19.472496 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:19.472998 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:19.473974 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:19.474173 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:19.474225 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:19.478863 (+ 4638us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 05:20:19.478864 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 05:20:19.478864 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 05:20:19.478871 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 05:20:19.480299 (+ 1428us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:20:19.480747 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:20:19.480748 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:20:19.480748 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:20:19.480805 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:20:19.481050 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:20:19.481051 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:20:19.481051 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:20:19.481148 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 05:20:19.481202 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0812 05:20:19.481491 (+ 289us) server_negotiation.cc:299] Negotiation successful
0812 05:20:19.481580 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260812 05:20:19.481884 32548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:19.470790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.62:42079 (local address 127.31.100.1:49099)
0812 05:20:19.470943 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0812 05:20:19.470978 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0812 05:20:19.471406 (+ 428us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:20:19.471598 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:20:19.471601 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:20:19.472224 (+ 623us) client_negotiation.cc:189] Negotiated authn=SASL
0812 05:20:19.472407 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:20:19.472411 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:20:19.473063 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:20:19.473064 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:20:19.473897 (+ 833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:20:19.473899 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:20:19.474245 (+ 346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:20:19.474246 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:20:19.474346 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:19.477348 (+ 3002us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 05:20:19.477356 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0812 05:20:19.478780 (+ 1424us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 05:20:19.480392 (+ 1612us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 05:20:19.480401 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 05:20:19.480403 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 05:20:19.480680 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 05:20:19.480860 (+ 180us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 05:20:19.480861 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 05:20:19.480861 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 05:20:19.480998 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 05:20:19.481212 (+ 214us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 05:20:19.481220 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 05:20:19.481342 (+ 122us) client_negotiation.cc:769] Sending connection context
0812 05:20:19.481428 (+ 86us) client_negotiation.cc:240] Negotiation successful
0812 05:20:19.481609 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":85,"threads_started":1}
I20260812 05:20:19.482342 32546 heartbeater.cc:347] Connected to a master server at 127.31.100.62:42079
I20260812 05:20:19.482450 32546 heartbeater.cc:464] Registering TS with master...
I20260812 05:20:19.482617 32546 heartbeater.cc:511] Master 127.31.100.62:42079 requested a full tablet report, sending...
I20260812 05:20:19.483129 32237 ts_manager.cc:194] Registered new tserver with Master: 613d489725344176868fd3db02b78b67 (127.31.100.1:36323)
I20260812 05:20:19.483937 32237 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.100.1@KRBTEST.COM'} at 127.31.100.1:49099
WARNING: no policy specified for kudu/127.31.100.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.100.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.100.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.2.keytab.
Entry for principal kudu/127.31.100.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.2.keytab.
I20260812 05:20:19.502009 32144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0waFvT/build/release/bin/kudu
/tmp/dist-test-task0waFvT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786512017613366-32144-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786512017613366-32144-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786512017613366-32144-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786512017613366-32144-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.100.2:0
--local_ip_for_outbound_sockets=127.31.100.2
--webserver_interface=127.31.100.2
--webserver_port=0
--tserver_master_addrs=127.31.100.62:42079
--keytab_file=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.2.keytab
--principal=kudu/127.31.100.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.100.20:36793
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786512017613366-32144-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786512017613366-32144-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786512017613366-32144-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 05:20:19.585111 32557 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 05:20:19.585306 32557 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 05:20:19.585340 32557 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 05:20:19.586557 32557 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 05:20:19.586602 32557 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 05:20:19.586637 32557 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.100.2
I20260812 05:20:19.588081 32557 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.100.20:36793
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786512017613366-32144-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786512017613366-32144-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.2.keytab
--principal=kudu/127.31.100.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.100.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786512017613366-32144-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.100.2
--webserver_port=0
--tserver_master_addrs=127.31.100.62:42079
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32557
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.100.2
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786512017613366-32144-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 04:20:34 UTC on ccycloud.quasar-phqdqp.root.comops.site
build id 4581871
I20260812 05:20:19.588315 32557 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 05:20:19.588538 32557 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 05:20:19.593328 32569 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 05:20:19.593366 32567 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 05:20:19.593616 32557 server_base.cc:1034] running on GCE node
W20260812 05:20:19.593310 32565 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 05:20:19.593871 32557 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 05:20:19.594096 32557 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 05:20:19.595235 32557 hybrid_clock.cc:648] HybridClock initialized: now 1786512019595221 us; error 27 us; skew 500 ppm
Aug 12 05:20:19 dist-test-slave-42z9 krb5kdc[32153](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512019, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.100.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 05:20:19.597605 32557 init.cc:377] Logged in from keytab as kudu/127.31.100.2@KRBTEST.COM (short username kudu)
I20260812 05:20:19.598011 32557 webserver.cc:460] Webserver started at http://127.31.100.2:42079/ using document root <none> and password file <none>
I20260812 05:20:19.598227 32557 fs_manager.cc:359] Metadata directory not provided
I20260812 05:20:19.598274 32557 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 05:20:19.598399 32557 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 05:20:19.599421 32557 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786512017613366-32144-0/minicluster-data/ts-1/data/instance:
uuid: "97a80bcaa1d14bbeb39d5902dae02771"
format_stamp: "Formatted at 2026-08-12 05:20:19 on dist-test-slave-42z9"
server_key: "b04fd0deb78cbade5311bcdc7cb7db8f"
server_key_iv: "076764a076ebb9763e9733b72d056af5"
server_key_version: "encryptionkey@0"
I20260812 05:20:19.599754 32557 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786512017613366-32144-0/minicluster-data/ts-1/wal/instance:
uuid: "97a80bcaa1d14bbeb39d5902dae02771"
format_stamp: "Formatted at 2026-08-12 05:20:19 on dist-test-slave-42z9"
server_key: "b04fd0deb78cbade5311bcdc7cb7db8f"
server_key_iv: "076764a076ebb9763e9733b72d056af5"
server_key_version: "encryptionkey@0"
I20260812 05:20:19.601042 32557 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260812 05:20:19.601701 32579 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 05:20:19.601848 32557 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260812 05:20:19.601930 32557 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786512017613366-32144-0/minicluster-data/ts-1/data,/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786512017613366-32144-0/minicluster-data/ts-1/wal
uuid: "97a80bcaa1d14bbeb39d5902dae02771"
format_stamp: "Formatted at 2026-08-12 05:20:19 on dist-test-slave-42z9"
server_key: "b04fd0deb78cbade5311bcdc7cb7db8f"
server_key_iv: "076764a076ebb9763e9733b72d056af5"
server_key_version: "encryptionkey@0"
I20260812 05:20:19.602002 32557 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786512017613366-32144-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786512017613366-32144-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786512017613366-32144-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 05:20:19.612107 32557 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 05:20:19.613716 32557 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 05:20:19.613837 32557 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 05:20:19.614019 32557 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 05:20:19.614283 32557 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 05:20:19.614315 32557 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 05:20:19.614341 32557 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 05:20:19.614364 32557 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 05:20:19.621755 32557 rpc_server.cc:307] RPC server started. Bound to: 127.31.100.2:43973
I20260812 05:20:19.621790 318 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.100.2:43973 every 8 connection(s)
I20260812 05:20:19.622145 32557 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786512017613366-32144-0/minicluster-data/ts-1/data/info.pb
I20260812 05:20:19.628856 32144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0waFvT/build/release/bin/kudu as pid 32557
I20260812 05:20:19.628939 32144 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786512017613366-32144-0/minicluster-data/ts-1/wal/instance
I20260812 05:20:19.629091 32144 external_mini_cluster.cc:1428] Setting key 9a65faf49da690f4793b96f6569df1a5
Aug 12 05:20:19 dist-test-slave-42z9 krb5kdc[32153](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512019, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.100.2@KRBTEST.COM for kudu/127.31.100.62@KRBTEST.COM
I20260812 05:20:19.636418 32291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:19.627211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.100.2:47331 (local address 127.31.100.62:42079)
0812 05:20:19.627269 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0812 05:20:19.627271 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 05:20:19.627280 (+ 9us) server_negotiation.cc:407] Connection header received
0812 05:20:19.627307 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:20:19.627307 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:20:19.627318 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:20:19.627366 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:20:19.628454 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:19.629077 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:19.630443 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:19.630702 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:19.630743 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:19.633150 (+ 2407us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 05:20:19.633151 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 05:20:19.633151 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 05:20:19.633159 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0812 05:20:19.634790 (+ 1631us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:20:19.635271 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:20:19.635273 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:20:19.635273 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:20:19.635338 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:20:19.635643 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:20:19.635644 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:20:19.635645 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:20:19.635791 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 05:20:19.635834 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0812 05:20:19.636230 (+ 396us) server_negotiation.cc:299] Negotiation successful
0812 05:20:19.636319 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260812 05:20:19.636827 324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:19.622981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.62:42079 (local address 127.31.100.2:47331)
0812 05:20:19.623158 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0812 05:20:19.623202 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0812 05:20:19.623870 (+ 668us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:20:19.627493 (+ 3623us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:20:19.627497 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:20:19.628127 (+ 630us) client_negotiation.cc:189] Negotiated authn=SASL
0812 05:20:19.628358 (+ 231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:20:19.628363 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:20:19.629179 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:20:19.629180 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:20:19.630343 (+ 1163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:20:19.630346 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:20:19.630830 (+ 484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:20:19.630831 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:20:19.630965 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:19.631494 (+ 529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 05:20:19.631505 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0812 05:20:19.633057 (+ 1552us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 05:20:19.634861 (+ 1804us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 05:20:19.634868 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 05:20:19.634871 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 05:20:19.635186 (+ 315us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 05:20:19.635404 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 05:20:19.635405 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 05:20:19.635405 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 05:20:19.635549 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 05:20:19.635882 (+ 333us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 05:20:19.635889 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 05:20:19.636059 (+ 170us) client_negotiation.cc:769] Sending connection context
0812 05:20:19.636191 (+ 132us) client_negotiation.cc:240] Negotiation successful
0812 05:20:19.636354 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":88,"threads_started":1}
I20260812 05:20:19.637422 319 heartbeater.cc:347] Connected to a master server at 127.31.100.62:42079
I20260812 05:20:19.637563 319 heartbeater.cc:464] Registering TS with master...
I20260812 05:20:19.637763 319 heartbeater.cc:511] Master 127.31.100.62:42079 requested a full tablet report, sending...
I20260812 05:20:19.638162 32237 ts_manager.cc:194] Registered new tserver with Master: 97a80bcaa1d14bbeb39d5902dae02771 (127.31.100.2:43973)
I20260812 05:20:19.638653 32237 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.100.2@KRBTEST.COM'} at 127.31.100.2:47331
WARNING: no policy specified for kudu/127.31.100.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.100.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.100.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.3.keytab.
Entry for principal kudu/127.31.100.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.3.keytab.
I20260812 05:20:19.662393 32144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0waFvT/build/release/bin/kudu
/tmp/dist-test-task0waFvT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786512017613366-32144-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786512017613366-32144-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786512017613366-32144-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786512017613366-32144-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.100.3:0
--local_ip_for_outbound_sockets=127.31.100.3
--webserver_interface=127.31.100.3
--webserver_port=0
--tserver_master_addrs=127.31.100.62:42079
--keytab_file=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.3.keytab
--principal=kudu/127.31.100.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.100.20:36793
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786512017613366-32144-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786512017613366-32144-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786512017613366-32144-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 05:20:19.745568 328 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 05:20:19.745760 328 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 05:20:19.745781 328 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 05:20:19.747108 328 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 05:20:19.747164 328 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 05:20:19.747200 328 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.100.3
I20260812 05:20:19.748693 328 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.100.20:36793
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786512017613366-32144-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786512017613366-32144-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.3.keytab
--principal=kudu/127.31.100.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.100.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786512017613366-32144-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.100.3
--webserver_port=0
--tserver_master_addrs=127.31.100.62:42079
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.328
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.100.3
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786512017613366-32144-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 04:20:34 UTC on ccycloud.quasar-phqdqp.root.comops.site
build id 4581871
I20260812 05:20:19.748898 328 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 05:20:19.749100 328 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 05:20:19.753845 337 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 05:20:19.753844 342 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 05:20:19.753898 339 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 05:20:19.754218 328 server_base.cc:1034] running on GCE node
I20260812 05:20:19.754410 328 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 05:20:19.754624 328 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 05:20:19.755753 328 hybrid_clock.cc:648] HybridClock initialized: now 1786512019755737 us; error 35 us; skew 500 ppm
Aug 12 05:20:19 dist-test-slave-42z9 krb5kdc[32153](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512019, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.100.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 05:20:19.758288 328 init.cc:377] Logged in from keytab as kudu/127.31.100.3@KRBTEST.COM (short username kudu)
I20260812 05:20:19.758721 328 webserver.cc:460] Webserver started at http://127.31.100.3:45395/ using document root <none> and password file <none>
I20260812 05:20:19.758925 328 fs_manager.cc:359] Metadata directory not provided
I20260812 05:20:19.758977 328 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 05:20:19.759101 328 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 05:20:19.760157 328 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786512017613366-32144-0/minicluster-data/ts-2/data/instance:
uuid: "90dd9eaf52a2452eb7708c96a583d97e"
format_stamp: "Formatted at 2026-08-12 05:20:19 on dist-test-slave-42z9"
server_key: "b75ce808db0a8efd31eec4bcf5bd9785"
server_key_iv: "d6c32806fd69227d57eb90c5a2f028f8"
server_key_version: "encryptionkey@0"
I20260812 05:20:19.760553 328 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786512017613366-32144-0/minicluster-data/ts-2/wal/instance:
uuid: "90dd9eaf52a2452eb7708c96a583d97e"
format_stamp: "Formatted at 2026-08-12 05:20:19 on dist-test-slave-42z9"
server_key: "b75ce808db0a8efd31eec4bcf5bd9785"
server_key_iv: "d6c32806fd69227d57eb90c5a2f028f8"
server_key_version: "encryptionkey@0"
I20260812 05:20:19.761816 328 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260812 05:20:19.762528 354 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 05:20:19.762703 328 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260812 05:20:19.762785 328 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786512017613366-32144-0/minicluster-data/ts-2/data,/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786512017613366-32144-0/minicluster-data/ts-2/wal
uuid: "90dd9eaf52a2452eb7708c96a583d97e"
format_stamp: "Formatted at 2026-08-12 05:20:19 on dist-test-slave-42z9"
server_key: "b75ce808db0a8efd31eec4bcf5bd9785"
server_key_iv: "d6c32806fd69227d57eb90c5a2f028f8"
server_key_version: "encryptionkey@0"
I20260812 05:20:19.762840 328 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786512017613366-32144-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786512017613366-32144-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786512017613366-32144-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 05:20:19.782696 328 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 05:20:19.784654 328 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 05:20:19.784807 328 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 05:20:19.785017 328 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 05:20:19.785323 328 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 05:20:19.785365 328 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 05:20:19.785394 328 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 05:20:19.785418 328 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 05:20:19.791966 328 rpc_server.cc:307] RPC server started. Bound to: 127.31.100.3:37077
I20260812 05:20:19.792049 564 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.100.3:37077 every 8 connection(s)
I20260812 05:20:19.792354 328 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786512017613366-32144-0/minicluster-data/ts-2/data/info.pb
I20260812 05:20:19.797794 32144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0waFvT/build/release/bin/kudu as pid 328
I20260812 05:20:19.797863 32144 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786512017613366-32144-0/minicluster-data/ts-2/wal/instance
I20260812 05:20:19.797991 32144 external_mini_cluster.cc:1428] Setting key 9d76c222f120a4d71bc4ee96df97bdaf
Aug 12 05:20:19 dist-test-slave-42z9 krb5kdc[32153](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512019, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.100.3@KRBTEST.COM for kudu/127.31.100.62@KRBTEST.COM
I20260812 05:20:19.800860 32291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:19.793139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.100.3:51105 (local address 127.31.100.62:42079)
0812 05:20:19.793184 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0812 05:20:19.793186 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 05:20:19.793367 (+ 181us) server_negotiation.cc:407] Connection header received
0812 05:20:19.793886 (+ 519us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:20:19.793887 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:20:19.793897 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:20:19.793952 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:20:19.794836 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:19.795331 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:19.796277 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:19.796446 (+ 169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:19.796487 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:19.798674 (+ 2187us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 05:20:19.798675 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 05:20:19.798675 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 05:20:19.798684 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0812 05:20:19.799842 (+ 1158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:20:19.800202 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:20:19.800203 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:20:19.800203 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:20:19.800250 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:20:19.800422 (+ 172us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:20:19.800423 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:20:19.800423 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:20:19.800516 (+ 93us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 05:20:19.800561 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0812 05:20:19.800743 (+ 182us) server_negotiation.cc:299] Negotiation successful
0812 05:20:19.800799 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260812 05:20:19.801040 569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:19.793079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.62:42079 (local address 127.31.100.3:51105)
0812 05:20:19.793249 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0812 05:20:19.793291 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0812 05:20:19.793802 (+ 511us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:20:19.794004 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:20:19.794007 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:20:19.794572 (+ 565us) client_negotiation.cc:189] Negotiated authn=SASL
0812 05:20:19.794760 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:20:19.794765 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:20:19.795410 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:20:19.795411 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:20:19.796215 (+ 804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:20:19.796218 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:20:19.796501 (+ 283us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:20:19.796502 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:20:19.796600 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:19.797072 (+ 472us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 05:20:19.797080 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0812 05:20:19.798569 (+ 1489us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 05:20:19.799907 (+ 1338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 05:20:19.799913 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 05:20:19.799914 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 05:20:19.800145 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 05:20:19.800294 (+ 149us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 05:20:19.800294 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 05:20:19.800295 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 05:20:19.800382 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 05:20:19.800557 (+ 175us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 05:20:19.800559 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 05:20:19.800659 (+ 100us) client_negotiation.cc:769] Sending connection context
0812 05:20:19.800731 (+ 72us) client_negotiation.cc:240] Negotiation successful
0812 05:20:19.800839 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":90,"threads_started":1}
I20260812 05:20:19.801483 565 heartbeater.cc:347] Connected to a master server at 127.31.100.62:42079
I20260812 05:20:19.801573 565 heartbeater.cc:464] Registering TS with master...
I20260812 05:20:19.801736 565 heartbeater.cc:511] Master 127.31.100.62:42079 requested a full tablet report, sending...
I20260812 05:20:19.802064 32237 ts_manager.cc:194] Registered new tserver with Master: 90dd9eaf52a2452eb7708c96a583d97e (127.31.100.3:37077)
I20260812 05:20:19.802408 32237 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.100.3@KRBTEST.COM'} at 127.31.100.3:51105
I20260812 05:20:19.809792 32144 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 12 05:20:19 dist-test-slave-42z9 krb5kdc[32153](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512019, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 12 05:20:19 dist-test-slave-42z9 krb5kdc[32153](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512019, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.31.100.1@KRBTEST.COM
I20260812 05:20:19.832378 578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:19.825715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57128 (local address 127.31.100.1:36323)
0812 05:20:19.825855 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0812 05:20:19.825860 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 05:20:19.825870 (+ 10us) server_negotiation.cc:407] Connection header received
0812 05:20:19.825894 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:20:19.825896 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:20:19.825908 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:20:19.825993 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:20:19.826500 (+ 507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:19.827215 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:19.827915 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:19.828222 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:19.828271 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:19.830273 (+ 2002us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 05:20:19.830280 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 05:20:19.830280 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 05:20:19.830286 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0812 05:20:19.831536 (+ 1250us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:20:19.831815 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:20:19.831818 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:20:19.831818 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:20:19.831864 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:20:19.831986 (+ 122us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:20:19.831987 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:20:19.831987 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:20:19.832094 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 05:20:19.832202 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0812 05:20:19.832234 (+ 32us) server_negotiation.cc:299] Negotiation successful
0812 05:20:19.832292 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107,"thread_start_us":65,"threads_started":1}
W20260812 05:20:19.832752 32402 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57128
Aug 12 05:20:19 dist-test-slave-42z9 krb5kdc[32153](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512019, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 12 05:20:19 dist-test-slave-42z9 krb5kdc[32153](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512019, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.100.1@KRBTEST.COM
I20260812 05:20:19.853565 578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:19.846994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57134 (local address 127.31.100.1:36323)
0812 05:20:19.847048 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0812 05:20:19.847049 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0812 05:20:19.847117 (+ 68us) server_negotiation.cc:407] Connection header received
0812 05:20:19.847154 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:20:19.847155 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:20:19.847164 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:20:19.847209 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:20:19.847713 (+ 504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:19.848187 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:19.848929 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:19.849137 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:19.849185 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:19.851341 (+ 2156us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 05:20:19.851343 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 05:20:19.851343 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 05:20:19.851349 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0812 05:20:19.852667 (+ 1318us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:20:19.852971 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:20:19.852971 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:20:19.852972 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:20:19.853016 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:20:19.853158 (+ 142us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:20:19.853158 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:20:19.853159 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:20:19.853249 (+ 90us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 05:20:19.853326 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0812 05:20:19.853392 (+ 66us) server_negotiation.cc:299] Negotiation successful
0812 05:20:19.853501 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260812 05:20:19.854058 32144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0waFvT/build/release/bin/kudu with pid 32333
I20260812 05:20:19.867218 32144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0waFvT/build/release/bin/kudu with pid 32557
I20260812 05:20:19.885731 32144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0waFvT/build/release/bin/kudu with pid 328
I20260812 05:20:19.896178 32144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0waFvT/build/release/bin/kudu with pid 32175
2026-08-12T05:20:19Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (2293 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786512017613366-32144-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 05:20:19 dist-test-slave-42z9 krb5kdc[594](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 05:20:19 dist-test-slave-42z9 krb5kdc[594](info): set up 2 sockets
Aug 12 05:20:19 dist-test-slave-42z9 krb5kdc[594](info): commencing operation
krb5kdc: starting...
W20260812 05:20:21.059433 32144 mini_kdc.cc:121] Time spent starting KDC: real 1.140s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786512017613366-32144-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786512017613366-32144-0/krb5kdc/test-user.keytab.
Aug 12 05:20:21 dist-test-slave-42z9 krb5kdc[594](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512021, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T05:20:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T05:20:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.100.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.100.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.100.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.62.keytab.
Entry for principal kudu/127.31.100.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.62.keytab.
I20260812 05:20:21.172017 32144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0waFvT/build/release/bin/kudu
/tmp/dist-test-task0waFvT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786512017613366-32144-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786512017613366-32144-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786512017613366-32144-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786512017613366-32144-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.100.62:33635
--webserver_interface=127.31.100.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.62.keytab
--principal=kudu/127.31.100.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.100.20:43227
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.100.62:33635
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786512017613366-32144-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786512017613366-32144-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786512017613366-32144-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 05:20:21.251211 618 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 05:20:21.251406 618 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 05:20:21.251446 618 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 05:20:21.252543 618 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 05:20:21.252583 618 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 05:20:21.252601 618 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 05:20:21.252619 618 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 05:20:21.252632 618 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 05:20:21.254133 618 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.100.20:43227
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786512017613366-32144-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786512017613366-32144-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.100.62:33635
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.62.keytab
--principal=kudu/127.31.100.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.100.62:33635
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786512017613366-32144-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.100.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.618
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786512017613366-32144-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 04:20:34 UTC on ccycloud.quasar-phqdqp.root.comops.site
build id 4581871
I20260812 05:20:21.254375 618 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 05:20:21.254582 618 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 05:20:21.259214 631 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 05:20:21.259208 634 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 05:20:21.259303 618 server_base.cc:1034] running on GCE node
W20260812 05:20:21.259208 637 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 05:20:21.259660 618 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 05:20:21.259930 618 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 05:20:21.261067 618 hybrid_clock.cc:648] HybridClock initialized: now 1786512021261050 us; error 33 us; skew 500 ppm
Aug 12 05:20:21 dist-test-slave-42z9 krb5kdc[594](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512021, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.100.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 05:20:21.263441 618 init.cc:377] Logged in from keytab as kudu/127.31.100.62@KRBTEST.COM (short username kudu)
I20260812 05:20:21.263860 618 webserver.cc:460] Webserver started at http://127.31.100.62:35077/ using document root <none> and password file <none>
I20260812 05:20:21.264050 618 fs_manager.cc:359] Metadata directory not provided
I20260812 05:20:21.264091 618 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 05:20:21.264214 618 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 05:20:21.265187 618 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786512017613366-32144-0/minicluster-data/master-0/data/instance:
uuid: "d7e8dbf6facb4ed9907df9a24cbe6bcb"
format_stamp: "Formatted at 2026-08-12 05:20:21 on dist-test-slave-42z9"
server_key: "d22d08e265f2be658b5b6ab7f9492e32"
server_key_iv: "d3188414a3db4255ef21dc8ce7f236c6"
server_key_version: "encryptionkey@0"
I20260812 05:20:21.265558 618 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786512017613366-32144-0/minicluster-data/master-0/wal/instance:
uuid: "d7e8dbf6facb4ed9907df9a24cbe6bcb"
format_stamp: "Formatted at 2026-08-12 05:20:21 on dist-test-slave-42z9"
server_key: "d22d08e265f2be658b5b6ab7f9492e32"
server_key_iv: "d3188414a3db4255ef21dc8ce7f236c6"
server_key_version: "encryptionkey@0"
I20260812 05:20:21.266754 618 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260812 05:20:21.267410 649 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 05:20:21.267570 618 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260812 05:20:21.267654 618 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786512017613366-32144-0/minicluster-data/master-0/data,/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786512017613366-32144-0/minicluster-data/master-0/wal
uuid: "d7e8dbf6facb4ed9907df9a24cbe6bcb"
format_stamp: "Formatted at 2026-08-12 05:20:21 on dist-test-slave-42z9"
server_key: "d22d08e265f2be658b5b6ab7f9492e32"
server_key_iv: "d3188414a3db4255ef21dc8ce7f236c6"
server_key_version: "encryptionkey@0"
I20260812 05:20:21.267709 618 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786512017613366-32144-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786512017613366-32144-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786512017613366-32144-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 05:20:21.294836 618 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 05:20:21.296798 618 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 05:20:21.296909 618 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 05:20:21.300572 618 rpc_server.cc:307] RPC server started. Bound to: 127.31.100.62:33635
I20260812 05:20:21.300614 738 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.100.62:33635 every 8 connection(s)
I20260812 05:20:21.300915 618 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786512017613366-32144-0/minicluster-data/master-0/data/info.pb
I20260812 05:20:21.301790 739 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 05:20:21.304054 739 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d7e8dbf6facb4ed9907df9a24cbe6bcb: Bootstrap starting.
I20260812 05:20:21.304610 739 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d7e8dbf6facb4ed9907df9a24cbe6bcb: Neither blocks nor log segments found. Creating new log.
I20260812 05:20:21.304883 739 log.cc:824] T 00000000000000000000000000000000 P d7e8dbf6facb4ed9907df9a24cbe6bcb: Log is configured to *not* fsync() on all Append() calls
I20260812 05:20:21.305549 739 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d7e8dbf6facb4ed9907df9a24cbe6bcb: No bootstrap required, opened a new log
I20260812 05:20:21.306317 32144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0waFvT/build/release/bin/kudu as pid 618
I20260812 05:20:21.306389 32144 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786512017613366-32144-0/minicluster-data/master-0/wal/instance
I20260812 05:20:21.306540 32144 external_mini_cluster.cc:1428] Setting key f80722c84fd8944fa171409dd3630418
I20260812 05:20:21.307332 739 raft_consensus.cc:348] T 00000000000000000000000000000000 P d7e8dbf6facb4ed9907df9a24cbe6bcb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7e8dbf6facb4ed9907df9a24cbe6bcb" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 33635 } }
I20260812 05:20:21.307471 739 raft_consensus.cc:374] T 00000000000000000000000000000000 P d7e8dbf6facb4ed9907df9a24cbe6bcb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 05:20:21.307504 739 raft_consensus.cc:729] T 00000000000000000000000000000000 P d7e8dbf6facb4ed9907df9a24cbe6bcb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7e8dbf6facb4ed9907df9a24cbe6bcb, State: Initialized, Role: FOLLOWER
I20260812 05:20:21.307624 739 consensus_queue.cc:260] T 00000000000000000000000000000000 P d7e8dbf6facb4ed9907df9a24cbe6bcb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7e8dbf6facb4ed9907df9a24cbe6bcb" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 33635 } }
I20260812 05:20:21.307695 739 raft_consensus.cc:388] T 00000000000000000000000000000000 P d7e8dbf6facb4ed9907df9a24cbe6bcb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 05:20:21.307726 739 raft_consensus.cc:482] T 00000000000000000000000000000000 P d7e8dbf6facb4ed9907df9a24cbe6bcb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 05:20:21.307770 739 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d7e8dbf6facb4ed9907df9a24cbe6bcb [term 0 FOLLOWER]: Advancing to term 1
I20260812 05:20:21.308276 739 raft_consensus.cc:504] T 00000000000000000000000000000000 P d7e8dbf6facb4ed9907df9a24cbe6bcb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7e8dbf6facb4ed9907df9a24cbe6bcb" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 33635 } }
I20260812 05:20:21.308372 739 leader_election.cc:302] T 00000000000000000000000000000000 P d7e8dbf6facb4ed9907df9a24cbe6bcb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d7e8dbf6facb4ed9907df9a24cbe6bcb; no voters:
I20260812 05:20:21.308528 739 leader_election.cc:288] T 00000000000000000000000000000000 P d7e8dbf6facb4ed9907df9a24cbe6bcb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 05:20:21.308578 750 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d7e8dbf6facb4ed9907df9a24cbe6bcb [term 1 FOLLOWER]: Leader election won for term 1
I20260812 05:20:21.308694 750 raft_consensus.cc:686] T 00000000000000000000000000000000 P d7e8dbf6facb4ed9907df9a24cbe6bcb [term 1 LEADER]: Becoming Leader. State: Replica: d7e8dbf6facb4ed9907df9a24cbe6bcb, State: Running, Role: LEADER
I20260812 05:20:21.308774 750 consensus_queue.cc:237] T 00000000000000000000000000000000 P d7e8dbf6facb4ed9907df9a24cbe6bcb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7e8dbf6facb4ed9907df9a24cbe6bcb" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 33635 } }
I20260812 05:20:21.308804 739 sys_catalog.cc:564] T 00000000000000000000000000000000 P d7e8dbf6facb4ed9907df9a24cbe6bcb [sys.catalog]: configured and running, proceeding with master startup.
I20260812 05:20:21.310004 751 sys_catalog.cc:455] T 00000000000000000000000000000000 P d7e8dbf6facb4ed9907df9a24cbe6bcb [sys.catalog]: SysCatalogTable state changed. Reason: New leader d7e8dbf6facb4ed9907df9a24cbe6bcb. Latest consensus state: current_term: 1 leader_uuid: "d7e8dbf6facb4ed9907df9a24cbe6bcb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7e8dbf6facb4ed9907df9a24cbe6bcb" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 33635 } } }
I20260812 05:20:21.310103 751 sys_catalog.cc:458] T 00000000000000000000000000000000 P d7e8dbf6facb4ed9907df9a24cbe6bcb [sys.catalog]: This master's current role is: LEADER
I20260812 05:20:21.310197 754 sys_catalog.cc:455] T 00000000000000000000000000000000 P d7e8dbf6facb4ed9907df9a24cbe6bcb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d7e8dbf6facb4ed9907df9a24cbe6bcb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7e8dbf6facb4ed9907df9a24cbe6bcb" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 33635 } } }
I20260812 05:20:21.310498 754 sys_catalog.cc:458] T 00000000000000000000000000000000 P d7e8dbf6facb4ed9907df9a24cbe6bcb [sys.catalog]: This master's current role is: LEADER
W20260812 05:20:21.310567 769 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d7e8dbf6facb4ed9907df9a24cbe6bcb: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 05:20:21.310628 769 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
Aug 12 05:20:21 dist-test-slave-42z9 krb5kdc[594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512021, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.100.62@KRBTEST.COM
I20260812 05:20:21.313602 772 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 05:20:21.313807 772 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 05:20:21.315791 772 catalog_manager.cc:1347] Generated new cluster ID: 5a8ddd6d5128486f930c8ace991c02e4
I20260812 05:20:21.315842 772 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 05:20:21.316681 749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:21.307086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49894 (local address 127.31.100.62:33635)
0812 05:20:21.307278 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0812 05:20:21.307286 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 05:20:21.307309 (+ 23us) server_negotiation.cc:407] Connection header received
0812 05:20:21.307398 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:20:21.307403 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:20:21.307444 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:20:21.307568 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:20:21.308311 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:21.309484 (+ 1173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:21.310516 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:21.311012 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:21.311113 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:21.313283 (+ 2170us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 05:20:21.313287 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 05:20:21.313288 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 05:20:21.313298 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0812 05:20:21.314857 (+ 1559us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:20:21.315263 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:20:21.315265 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:20:21.315265 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:20:21.315536 (+ 271us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:20:21.315773 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:20:21.315774 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:20:21.315775 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:20:21.316084 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 05:20:21.316237 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0812 05:20:21.316355 (+ 118us) server_negotiation.cc:299] Negotiation successful
0812 05:20:21.316491 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140,"thread_start_us":100,"threads_started":1}
I20260812 05:20:21.334020 772 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 05:20:21.334708 772 catalog_manager.cc:1504] Loading token signing keys...
I20260812 05:20:21.345069 772 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d7e8dbf6facb4ed9907df9a24cbe6bcb: Generated new TSK 0
I20260812 05:20:21.345233 772 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.31.100.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.100.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.100.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.1.keytab.
Entry for principal kudu/127.31.100.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.1.keytab.
I20260812 05:20:21.375995 32144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0waFvT/build/release/bin/kudu
/tmp/dist-test-task0waFvT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786512017613366-32144-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786512017613366-32144-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786512017613366-32144-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786512017613366-32144-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.100.1:0
--local_ip_for_outbound_sockets=127.31.100.1
--webserver_interface=127.31.100.1
--webserver_port=0
--tserver_master_addrs=127.31.100.62:33635
--keytab_file=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.1.keytab
--principal=kudu/127.31.100.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.100.20:43227
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786512017613366-32144-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786512017613366-32144-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786512017613366-32144-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 05:20:21.455309 780 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 05:20:21.455503 780 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 05:20:21.455526 780 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 05:20:21.456660 780 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 05:20:21.456703 780 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 05:20:21.456727 780 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.100.1
I20260812 05:20:21.458263 780 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.100.20:43227
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786512017613366-32144-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786512017613366-32144-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.1.keytab
--principal=kudu/127.31.100.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.100.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786512017613366-32144-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.100.1
--webserver_port=0
--tserver_master_addrs=127.31.100.62:33635
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.780
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.100.1
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786512017613366-32144-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 04:20:34 UTC on ccycloud.quasar-phqdqp.root.comops.site
build id 4581871
I20260812 05:20:21.458510 780 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 05:20:21.458724 780 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 05:20:21.463582 795 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 05:20:21.463603 786 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 05:20:21.463582 791 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 05:20:21.463639 780 server_base.cc:1034] running on GCE node
I20260812 05:20:21.463980 780 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 05:20:21.464210 780 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 05:20:21.465355 780 hybrid_clock.cc:648] HybridClock initialized: now 1786512021465344 us; error 28 us; skew 500 ppm
Aug 12 05:20:21 dist-test-slave-42z9 krb5kdc[594](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512021, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.100.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 05:20:21.467736 780 init.cc:377] Logged in from keytab as kudu/127.31.100.1@KRBTEST.COM (short username kudu)
I20260812 05:20:21.468108 780 webserver.cc:460] Webserver started at http://127.31.100.1:45937/ using document root <none> and password file <none>
I20260812 05:20:21.468302 780 fs_manager.cc:359] Metadata directory not provided
I20260812 05:20:21.468339 780 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 05:20:21.468436 780 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 05:20:21.469379 780 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786512017613366-32144-0/minicluster-data/ts-0/data/instance:
uuid: "4e067bcd6df94544b2d2a9f06efb3948"
format_stamp: "Formatted at 2026-08-12 05:20:21 on dist-test-slave-42z9"
server_key: "3b31a943d149a29a1dd82d202e781d6b"
server_key_iv: "0b802682a1d9602b9d64bf7265288591"
server_key_version: "encryptionkey@0"
I20260812 05:20:21.469722 780 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786512017613366-32144-0/minicluster-data/ts-0/wal/instance:
uuid: "4e067bcd6df94544b2d2a9f06efb3948"
format_stamp: "Formatted at 2026-08-12 05:20:21 on dist-test-slave-42z9"
server_key: "3b31a943d149a29a1dd82d202e781d6b"
server_key_iv: "0b802682a1d9602b9d64bf7265288591"
server_key_version: "encryptionkey@0"
I20260812 05:20:21.470996 780 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260812 05:20:21.471634 804 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 05:20:21.471777 780 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260812 05:20:21.471854 780 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786512017613366-32144-0/minicluster-data/ts-0/data,/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786512017613366-32144-0/minicluster-data/ts-0/wal
uuid: "4e067bcd6df94544b2d2a9f06efb3948"
format_stamp: "Formatted at 2026-08-12 05:20:21 on dist-test-slave-42z9"
server_key: "3b31a943d149a29a1dd82d202e781d6b"
server_key_iv: "0b802682a1d9602b9d64bf7265288591"
server_key_version: "encryptionkey@0"
I20260812 05:20:21.471920 780 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786512017613366-32144-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786512017613366-32144-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786512017613366-32144-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 05:20:21.484117 780 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 05:20:21.486090 780 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 05:20:21.486228 780 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 05:20:21.486447 780 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 05:20:21.486750 780 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 05:20:21.486784 780 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 05:20:21.486811 780 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 05:20:21.486836 780 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 05:20:21.492172 780 rpc_server.cc:307] RPC server started. Bound to: 127.31.100.1:33241
I20260812 05:20:21.492234 1005 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.100.1:33241 every 8 connection(s)
I20260812 05:20:21.492512 780 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786512017613366-32144-0/minicluster-data/ts-0/data/info.pb
Aug 12 05:20:21 dist-test-slave-42z9 krb5kdc[594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512021, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.100.1@KRBTEST.COM for kudu/127.31.100.62@KRBTEST.COM
I20260812 05:20:21.500555 32144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0waFvT/build/release/bin/kudu as pid 780
I20260812 05:20:21.500655 32144 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786512017613366-32144-0/minicluster-data/ts-0/wal/instance
I20260812 05:20:21.500826 32144 external_mini_cluster.cc:1428] Setting key 111b8369fb6388b037f2070a04523741
I20260812 05:20:21.501405 749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:21.493192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.100.1:47739 (local address 127.31.100.62:33635)
0812 05:20:21.493235 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0812 05:20:21.493236 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0812 05:20:21.493471 (+ 235us) server_negotiation.cc:407] Connection header received
0812 05:20:21.494511 (+ 1040us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:20:21.494512 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:20:21.494523 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:20:21.494575 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:20:21.495146 (+ 571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:21.495666 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:21.496645 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:21.496844 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:21.496894 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:21.498860 (+ 1966us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 05:20:21.498862 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 05:20:21.498862 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 05:20:21.498870 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0812 05:20:21.500134 (+ 1264us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:20:21.500464 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:20:21.500465 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:20:21.500465 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:20:21.500513 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:20:21.500762 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:20:21.500763 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:20:21.500763 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:20:21.500876 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 05:20:21.500942 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0812 05:20:21.501244 (+ 302us) server_negotiation.cc:299] Negotiation successful
0812 05:20:21.501319 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260812 05:20:21.501629 1010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:21.493154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.62:33635 (local address 127.31.100.1:47739)
0812 05:20:21.493316 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0812 05:20:21.493357 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0812 05:20:21.494412 (+ 1055us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:20:21.494637 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:20:21.494640 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:20:21.494898 (+ 258us) client_negotiation.cc:189] Negotiated authn=SASL
0812 05:20:21.495082 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:20:21.495086 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:20:21.495730 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:20:21.495731 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:20:21.496569 (+ 838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:20:21.496572 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:20:21.496901 (+ 329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:20:21.496902 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:20:21.497003 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:21.497433 (+ 430us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 05:20:21.497475 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0812 05:20:21.498776 (+ 1301us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 05:20:21.500198 (+ 1422us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 05:20:21.500201 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 05:20:21.500203 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 05:20:21.500400 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 05:20:21.500561 (+ 161us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 05:20:21.500562 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 05:20:21.500562 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 05:20:21.500696 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 05:20:21.500942 (+ 246us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 05:20:21.500950 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 05:20:21.501109 (+ 159us) client_negotiation.cc:769] Sending connection context
0812 05:20:21.501217 (+ 108us) client_negotiation.cc:240] Negotiation successful
0812 05:20:21.501378 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":78,"threads_started":1}
I20260812 05:20:21.502144 1006 heartbeater.cc:347] Connected to a master server at 127.31.100.62:33635
I20260812 05:20:21.502234 1006 heartbeater.cc:464] Registering TS with master...
I20260812 05:20:21.502403 1006 heartbeater.cc:511] Master 127.31.100.62:33635 requested a full tablet report, sending...
I20260812 05:20:21.502892 675 ts_manager.cc:194] Registered new tserver with Master: 4e067bcd6df94544b2d2a9f06efb3948 (127.31.100.1:33241)
I20260812 05:20:21.503592 675 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.100.1@KRBTEST.COM'} at 127.31.100.1:47739
WARNING: no policy specified for kudu/127.31.100.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.100.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.100.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.2.keytab.
Entry for principal kudu/127.31.100.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.2.keytab.
I20260812 05:20:21.525120 32144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0waFvT/build/release/bin/kudu
/tmp/dist-test-task0waFvT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786512017613366-32144-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786512017613366-32144-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786512017613366-32144-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786512017613366-32144-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.100.2:0
--local_ip_for_outbound_sockets=127.31.100.2
--webserver_interface=127.31.100.2
--webserver_port=0
--tserver_master_addrs=127.31.100.62:33635
--keytab_file=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.2.keytab
--principal=kudu/127.31.100.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.100.20:43227
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786512017613366-32144-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786512017613366-32144-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786512017613366-32144-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 05:20:21.603946 1015 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 05:20:21.604142 1015 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 05:20:21.604166 1015 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 05:20:21.605227 1015 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 05:20:21.605260 1015 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 05:20:21.605285 1015 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.100.2
I20260812 05:20:21.606715 1015 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.100.20:43227
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786512017613366-32144-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786512017613366-32144-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.2.keytab
--principal=kudu/127.31.100.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.100.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786512017613366-32144-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.100.2
--webserver_port=0
--tserver_master_addrs=127.31.100.62:33635
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1015
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.100.2
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786512017613366-32144-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 04:20:34 UTC on ccycloud.quasar-phqdqp.root.comops.site
build id 4581871
I20260812 05:20:21.606891 1015 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 05:20:21.607074 1015 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 05:20:21.611975 1028 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 05:20:21.612035 1022 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 05:20:21.612013 1024 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 05:20:21.612167 1015 server_base.cc:1034] running on GCE node
I20260812 05:20:21.612383 1015 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 05:20:21.612602 1015 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 05:20:21.613729 1015 hybrid_clock.cc:648] HybridClock initialized: now 1786512021613711 us; error 35 us; skew 500 ppm
Aug 12 05:20:21 dist-test-slave-42z9 krb5kdc[594](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512021, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.100.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 05:20:21.616129 1015 init.cc:377] Logged in from keytab as kudu/127.31.100.2@KRBTEST.COM (short username kudu)
I20260812 05:20:21.616544 1015 webserver.cc:460] Webserver started at http://127.31.100.2:46691/ using document root <none> and password file <none>
I20260812 05:20:21.616735 1015 fs_manager.cc:359] Metadata directory not provided
I20260812 05:20:21.616777 1015 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 05:20:21.616909 1015 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 05:20:21.617957 1015 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786512017613366-32144-0/minicluster-data/ts-1/data/instance:
uuid: "2907452cbb30499cbf4c41c42ae48edc"
format_stamp: "Formatted at 2026-08-12 05:20:21 on dist-test-slave-42z9"
server_key: "114832cd2031d0bce3d3c4de84a9e4ec"
server_key_iv: "d019fcb8dce3a7ae5811857ad0cebd15"
server_key_version: "encryptionkey@0"
I20260812 05:20:21.618273 1015 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786512017613366-32144-0/minicluster-data/ts-1/wal/instance:
uuid: "2907452cbb30499cbf4c41c42ae48edc"
format_stamp: "Formatted at 2026-08-12 05:20:21 on dist-test-slave-42z9"
server_key: "114832cd2031d0bce3d3c4de84a9e4ec"
server_key_iv: "d019fcb8dce3a7ae5811857ad0cebd15"
server_key_version: "encryptionkey@0"
I20260812 05:20:21.619561 1015 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260812 05:20:21.620247 1036 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 05:20:21.620441 1015 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260812 05:20:21.620519 1015 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786512017613366-32144-0/minicluster-data/ts-1/data,/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786512017613366-32144-0/minicluster-data/ts-1/wal
uuid: "2907452cbb30499cbf4c41c42ae48edc"
format_stamp: "Formatted at 2026-08-12 05:20:21 on dist-test-slave-42z9"
server_key: "114832cd2031d0bce3d3c4de84a9e4ec"
server_key_iv: "d019fcb8dce3a7ae5811857ad0cebd15"
server_key_version: "encryptionkey@0"
I20260812 05:20:21.620570 1015 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786512017613366-32144-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786512017613366-32144-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786512017613366-32144-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 05:20:21.652832 1015 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 05:20:21.655048 1015 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 05:20:21.655164 1015 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 05:20:21.655355 1015 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 05:20:21.655627 1015 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 05:20:21.655658 1015 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 05:20:21.655678 1015 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 05:20:21.655694 1015 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 05:20:21.661072 1015 rpc_server.cc:307] RPC server started. Bound to: 127.31.100.2:38737
I20260812 05:20:21.661139 1221 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.100.2:38737 every 8 connection(s)
I20260812 05:20:21.661410 1015 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786512017613366-32144-0/minicluster-data/ts-1/data/info.pb
I20260812 05:20:21.662674 32144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0waFvT/build/release/bin/kudu as pid 1015
I20260812 05:20:21.662752 32144 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786512017613366-32144-0/minicluster-data/ts-1/wal/instance
I20260812 05:20:21.662899 32144 external_mini_cluster.cc:1428] Setting key 3b6218e70a1bfa96c9f9eef4ae83cec6
Aug 12 05:20:21 dist-test-slave-42z9 krb5kdc[594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512021, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.100.2@KRBTEST.COM for kudu/127.31.100.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.31.100.3@KRBTEST.COM; defaulting to no policy
I20260812 05:20:21.672156 749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:21.662098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.100.2:52969 (local address 127.31.100.62:33635)
0812 05:20:21.662158 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0812 05:20:21.662160 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 05:20:21.662350 (+ 190us) server_negotiation.cc:407] Connection header received
0812 05:20:21.663391 (+ 1041us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:20:21.663392 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:20:21.663409 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:20:21.663468 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:20:21.664079 (+ 611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:21.664576 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:21.665588 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:21.665795 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:21.665850 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:21.669431 (+ 3581us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 05:20:21.669433 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 05:20:21.669433 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 05:20:21.669463 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0812 05:20:21.670945 (+ 1482us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:20:21.671327 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:20:21.671329 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:20:21.671329 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:20:21.671385 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:20:21.671598 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:20:21.671599 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:20:21.671599 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:20:21.671732 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 05:20:21.671803 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0812 05:20:21.672025 (+ 222us) server_negotiation.cc:299] Negotiation successful
0812 05:20:21.672091 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260812 05:20:21.672317 1225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:21.662079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.62:33635 (local address 127.31.100.2:52969)
0812 05:20:21.662240 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0812 05:20:21.662274 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0812 05:20:21.663284 (+ 1010us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:20:21.663538 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:20:21.663542 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:20:21.663792 (+ 250us) client_negotiation.cc:189] Negotiated authn=SASL
0812 05:20:21.663989 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:20:21.663993 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:20:21.664656 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:20:21.664657 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:20:21.665507 (+ 850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:20:21.665509 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:20:21.665861 (+ 352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:20:21.665862 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:20:21.665992 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:21.667746 (+ 1754us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 05:20:21.667757 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0812 05:20:21.669332 (+ 1575us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 05:20:21.671020 (+ 1688us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 05:20:21.671026 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 05:20:21.671028 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 05:20:21.671237 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 05:20:21.671442 (+ 205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 05:20:21.671442 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 05:20:21.671443 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 05:20:21.671538 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 05:20:21.671792 (+ 254us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 05:20:21.671797 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 05:20:21.671911 (+ 114us) client_negotiation.cc:769] Sending connection context
0812 05:20:21.671998 (+ 87us) client_negotiation.cc:240] Negotiation successful
0812 05:20:21.672118 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":83,"threads_started":1}
I20260812 05:20:21.672783 1222 heartbeater.cc:347] Connected to a master server at 127.31.100.62:33635
I20260812 05:20:21.672869 1222 heartbeater.cc:464] Registering TS with master...
I20260812 05:20:21.673049 1222 heartbeater.cc:511] Master 127.31.100.62:33635 requested a full tablet report, sending...
I20260812 05:20:21.673398 675 ts_manager.cc:194] Registered new tserver with Master: 2907452cbb30499cbf4c41c42ae48edc (127.31.100.2:38737)
I20260812 05:20:21.673837 675 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.100.2@KRBTEST.COM'} at 127.31.100.2:52969
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.100.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.100.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.3.keytab.
Entry for principal kudu/127.31.100.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.3.keytab.
I20260812 05:20:21.687942 32144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0waFvT/build/release/bin/kudu
/tmp/dist-test-task0waFvT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786512017613366-32144-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786512017613366-32144-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786512017613366-32144-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786512017613366-32144-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.100.3:0
--local_ip_for_outbound_sockets=127.31.100.3
--webserver_interface=127.31.100.3
--webserver_port=0
--tserver_master_addrs=127.31.100.62:33635
--keytab_file=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.3.keytab
--principal=kudu/127.31.100.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.100.20:43227
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786512017613366-32144-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786512017613366-32144-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786512017613366-32144-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 05:20:21.768366 1229 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 05:20:21.768554 1229 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 05:20:21.768575 1229 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 05:20:21.769706 1229 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 05:20:21.769744 1229 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 05:20:21.769769 1229 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.100.3
I20260812 05:20:21.771230 1229 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.100.20:43227
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786512017613366-32144-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786512017613366-32144-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.3.keytab
--principal=kudu/127.31.100.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.100.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786512017613366-32144-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.100.3
--webserver_port=0
--tserver_master_addrs=127.31.100.62:33635
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1229
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.100.3
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786512017613366-32144-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 04:20:34 UTC on ccycloud.quasar-phqdqp.root.comops.site
build id 4581871
I20260812 05:20:21.771451 1229 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 05:20:21.771657 1229 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 05:20:21.776419 1245 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 05:20:21.776553 1229 server_base.cc:1034] running on GCE node
W20260812 05:20:21.776444 1244 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 05:20:21.776443 1247 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 05:20:21.776809 1229 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 05:20:21.777036 1229 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 05:20:21.778172 1229 hybrid_clock.cc:648] HybridClock initialized: now 1786512021778156 us; error 27 us; skew 500 ppm
Aug 12 05:20:21 dist-test-slave-42z9 krb5kdc[594](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512021, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.100.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 05:20:21.780510 1229 init.cc:377] Logged in from keytab as kudu/127.31.100.3@KRBTEST.COM (short username kudu)
I20260812 05:20:21.780898 1229 webserver.cc:460] Webserver started at http://127.31.100.3:35319/ using document root <none> and password file <none>
I20260812 05:20:21.781087 1229 fs_manager.cc:359] Metadata directory not provided
I20260812 05:20:21.781128 1229 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 05:20:21.781241 1229 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 05:20:21.782266 1229 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786512017613366-32144-0/minicluster-data/ts-2/data/instance:
uuid: "06646753b0e048e187a5cb4fda7858e8"
format_stamp: "Formatted at 2026-08-12 05:20:21 on dist-test-slave-42z9"
server_key: "10912a6fd77eb8f1196efcd1927fa521"
server_key_iv: "d40b6bd8de61a6823495a4d7fac2a108"
server_key_version: "encryptionkey@0"
I20260812 05:20:21.782578 1229 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786512017613366-32144-0/minicluster-data/ts-2/wal/instance:
uuid: "06646753b0e048e187a5cb4fda7858e8"
format_stamp: "Formatted at 2026-08-12 05:20:21 on dist-test-slave-42z9"
server_key: "10912a6fd77eb8f1196efcd1927fa521"
server_key_iv: "d40b6bd8de61a6823495a4d7fac2a108"
server_key_version: "encryptionkey@0"
I20260812 05:20:21.783886 1229 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260812 05:20:21.784621 1257 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 05:20:21.784798 1229 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260812 05:20:21.784883 1229 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786512017613366-32144-0/minicluster-data/ts-2/data,/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786512017613366-32144-0/minicluster-data/ts-2/wal
uuid: "06646753b0e048e187a5cb4fda7858e8"
format_stamp: "Formatted at 2026-08-12 05:20:21 on dist-test-slave-42z9"
server_key: "10912a6fd77eb8f1196efcd1927fa521"
server_key_iv: "d40b6bd8de61a6823495a4d7fac2a108"
server_key_version: "encryptionkey@0"
I20260812 05:20:21.784933 1229 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786512017613366-32144-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786512017613366-32144-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786512017613366-32144-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 05:20:21.802592 1229 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 05:20:21.804275 1229 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 05:20:21.804381 1229 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 05:20:21.804574 1229 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 05:20:21.804826 1229 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 05:20:21.804854 1229 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 05:20:21.804874 1229 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 05:20:21.804888 1229 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 05:20:21.810623 1229 rpc_server.cc:307] RPC server started. Bound to: 127.31.100.3:41909
I20260812 05:20:21.810734 1451 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.100.3:41909 every 8 connection(s)
I20260812 05:20:21.810983 1229 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786512017613366-32144-0/minicluster-data/ts-2/data/info.pb
I20260812 05:20:21.812769 32144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0waFvT/build/release/bin/kudu as pid 1229
I20260812 05:20:21.812862 32144 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786512017613366-32144-0/minicluster-data/ts-2/wal/instance
I20260812 05:20:21.813028 32144 external_mini_cluster.cc:1428] Setting key 3abb0045fd5492db3344d6fbb8558f0b
Aug 12 05:20:21 dist-test-slave-42z9 krb5kdc[594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512021, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.100.3@KRBTEST.COM for kudu/127.31.100.62@KRBTEST.COM
I20260812 05:20:21.819705 749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:21.811583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.100.3:33107 (local address 127.31.100.62:33635)
0812 05:20:21.811623 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0812 05:20:21.811625 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 05:20:21.811809 (+ 184us) server_negotiation.cc:407] Connection header received
0812 05:20:21.812756 (+ 947us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:20:21.812757 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:20:21.812770 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:20:21.812820 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:20:21.813576 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:21.814152 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:21.815099 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:21.815295 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:21.815349 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:21.817274 (+ 1925us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 05:20:21.817275 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 05:20:21.817275 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 05:20:21.817282 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 05:20:21.818497 (+ 1215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:20:21.818849 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:20:21.818850 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:20:21.818850 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:20:21.818904 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:20:21.819116 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:20:21.819117 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:20:21.819118 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:20:21.819225 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 05:20:21.819280 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0812 05:20:21.819543 (+ 263us) server_negotiation.cc:299] Negotiation successful
0812 05:20:21.819618 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260812 05:20:21.819854 1454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:21.811556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.62:33635 (local address 127.31.100.3:33107)
0812 05:20:21.811701 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0812 05:20:21.811736 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0812 05:20:21.812660 (+ 924us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:20:21.812886 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:20:21.812890 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:20:21.813212 (+ 322us) client_negotiation.cc:189] Negotiated authn=SASL
0812 05:20:21.813490 (+ 278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:20:21.813497 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:20:21.814231 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:20:21.814232 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:20:21.815022 (+ 790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:20:21.815024 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:20:21.815360 (+ 336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:20:21.815361 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:20:21.815495 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:21.815920 (+ 425us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 05:20:21.815928 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0812 05:20:21.817192 (+ 1264us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 05:20:21.818568 (+ 1376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 05:20:21.818575 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 05:20:21.818577 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 05:20:21.818785 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 05:20:21.818956 (+ 171us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 05:20:21.818957 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 05:20:21.818957 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 05:20:21.819061 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 05:20:21.819278 (+ 217us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 05:20:21.819284 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 05:20:21.819420 (+ 136us) client_negotiation.cc:769] Sending connection context
0812 05:20:21.819511 (+ 91us) client_negotiation.cc:240] Negotiation successful
0812 05:20:21.819657 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":73,"threads_started":1}
I20260812 05:20:21.820266 1452 heartbeater.cc:347] Connected to a master server at 127.31.100.62:33635
I20260812 05:20:21.820350 1452 heartbeater.cc:464] Registering TS with master...
I20260812 05:20:21.820503 1452 heartbeater.cc:511] Master 127.31.100.62:33635 requested a full tablet report, sending...
I20260812 05:20:21.820827 675 ts_manager.cc:194] Registered new tserver with Master: 06646753b0e048e187a5cb4fda7858e8 (127.31.100.3:41909)
I20260812 05:20:21.821166 675 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.100.3@KRBTEST.COM'} at 127.31.100.3:33107
I20260812 05:20:21.824607 32144 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 05:20:21.830854 749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:21.825997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49908 (local address 127.31.100.62:33635)
0812 05:20:21.826043 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0812 05:20:21.826045 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 05:20:21.826104 (+ 59us) server_negotiation.cc:407] Connection header received
0812 05:20:21.826150 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:20:21.826150 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:20:21.826157 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:20:21.826200 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:20:21.826675 (+ 475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:21.827106 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:21.827715 (+ 609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:21.827881 (+ 166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:21.827922 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:21.828892 (+ 970us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 05:20:21.828893 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 05:20:21.828893 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 05:20:21.828898 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0812 05:20:21.829945 (+ 1047us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:20:21.830255 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:20:21.830256 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:20:21.830256 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:20:21.830314 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:20:21.830467 (+ 153us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:20:21.830467 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:20:21.830467 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:20:21.830592 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 05:20:21.830664 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0812 05:20:21.830736 (+ 72us) server_negotiation.cc:299] Negotiation successful
0812 05:20:21.830797 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260812 05:20:21.832638 675 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49908:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 05:20:21.834180 675 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 05:20:21.840161 1465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:21.836222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.2:38737 (local address 127.0.0.1:52884)
0812 05:20:21.836465 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0812 05:20:21.836508 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0812 05:20:21.836572 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:20:21.836926 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:20:21.836930 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:20:21.836933 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 05:20:21.836995 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:20:21.836997 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:20:21.837983 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:20:21.837984 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:20:21.839129 (+ 1145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:20:21.839132 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:20:21.839872 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:20:21.839872 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:20:21.839957 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:21.839971 (+ 14us) client_negotiation.cc:769] Sending connection context
0812 05:20:21.840032 (+ 61us) client_negotiation.cc:240] Negotiation successful
0812 05:20:21.840087 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":52,"threads_started":1}
I20260812 05:20:21.840173 1468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:21.836387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52884 (local address 127.31.100.2:38737)
0812 05:20:21.836768 (+ 381us) server_negotiation.cc:206] Beginning negotiation
0812 05:20:21.836771 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 05:20:21.836781 (+ 10us) server_negotiation.cc:407] Connection header received
0812 05:20:21.836820 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:20:21.836821 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:20:21.836836 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:20:21.836911 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 05:20:21.837053 (+ 142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:21.837897 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:21.839221 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:21.839801 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:21.839858 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:21.839914 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0812 05:20:21.840052 (+ 138us) server_negotiation.cc:299] Negotiation successful
0812 05:20:21.840091 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":347,"thread_start_us":114,"threads_started":1}
I20260812 05:20:21.840269 1464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:21.836222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.3:41909 (local address 127.0.0.1:56434)
0812 05:20:21.836591 (+ 369us) negotiation.cc:107] Waiting for socket to connect
0812 05:20:21.836601 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0812 05:20:21.836654 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:20:21.837047 (+ 393us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:20:21.837048 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:20:21.837050 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 05:20:21.837079 (+ 29us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:20:21.837080 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:20:21.838366 (+ 1286us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:20:21.838367 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:20:21.839140 (+ 773us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:20:21.839143 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:20:21.840036 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:20:21.840037 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:20:21.840101 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:21.840105 (+ 4us) client_negotiation.cc:769] Sending connection context
0812 05:20:21.840152 (+ 47us) client_negotiation.cc:240] Negotiation successful
0812 05:20:21.840194 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":354,"mutex_wait_us":4,"thread_start_us":48,"threads_started":1}
I20260812 05:20:21.840451 1466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:21.836372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56434 (local address 127.31.100.3:41909)
0812 05:20:21.836834 (+ 462us) server_negotiation.cc:206] Beginning negotiation
0812 05:20:21.836839 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 05:20:21.836846 (+ 7us) server_negotiation.cc:407] Connection header received
0812 05:20:21.836889 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:20:21.836891 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:20:21.836906 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:20:21.836972 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 05:20:21.837148 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:21.838230 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:21.839228 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:21.839883 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:21.839949 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:21.840000 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0812 05:20:21.840337 (+ 337us) server_negotiation.cc:299] Negotiation successful
0812 05:20:21.840376 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":428,"thread_start_us":122,"threads_started":1}
I20260812 05:20:21.840529 1463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:21.836253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.1:33241 (local address 127.0.0.1:57876)
0812 05:20:21.836670 (+ 417us) negotiation.cc:107] Waiting for socket to connect
0812 05:20:21.836677 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0812 05:20:21.836717 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:20:21.836974 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:20:21.836975 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:20:21.836977 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 05:20:21.837033 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:20:21.837035 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:20:21.838337 (+ 1302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:20:21.838338 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:20:21.839422 (+ 1084us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:20:21.839424 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:20:21.840312 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:20:21.840313 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:20:21.840385 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:21.840388 (+ 3us) client_negotiation.cc:769] Sending connection context
0812 05:20:21.840437 (+ 49us) client_negotiation.cc:240] Negotiation successful
0812 05:20:21.840473 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":407,"thread_start_us":52,"threads_started":1}
I20260812 05:20:21.840751 1469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:21.836487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57876 (local address 127.31.100.1:33241)
0812 05:20:21.836665 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0812 05:20:21.836670 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 05:20:21.836738 (+ 68us) server_negotiation.cc:407] Connection header received
0812 05:20:21.836787 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:20:21.836790 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:20:21.836806 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:20:21.836877 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 05:20:21.837110 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:21.838230 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:21.839502 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:21.840175 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:21.840229 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:21.840278 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0812 05:20:21.840633 (+ 355us) server_negotiation.cc:299] Negotiation successful
0812 05:20:21.840678 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":72,"threads_started":1}
I20260812 05:20:21.840979 1114 tablet_service.cc:1467] Processing CreateTablet for tablet d99ba4dcbe7f46f38b1bfeb83add08e3 (DEFAULT_TABLE table=test-table [id=c23ce01217ec4145bc49bcde50ed1c5a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 05:20:21.841219 1340 tablet_service.cc:1467] Processing CreateTablet for tablet d99ba4dcbe7f46f38b1bfeb83add08e3 (DEFAULT_TABLE table=test-table [id=c23ce01217ec4145bc49bcde50ed1c5a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 05:20:21.841349 1114 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d99ba4dcbe7f46f38b1bfeb83add08e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 05:20:21.841369 878 tablet_service.cc:1467] Processing CreateTablet for tablet d99ba4dcbe7f46f38b1bfeb83add08e3 (DEFAULT_TABLE table=test-table [id=c23ce01217ec4145bc49bcde50ed1c5a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 05:20:21.841603 1340 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d99ba4dcbe7f46f38b1bfeb83add08e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 05:20:21.841686 878 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d99ba4dcbe7f46f38b1bfeb83add08e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 05:20:21.844048 1470 tablet_bootstrap.cc:492] T d99ba4dcbe7f46f38b1bfeb83add08e3 P 2907452cbb30499cbf4c41c42ae48edc: Bootstrap starting.
I20260812 05:20:21.844832 1470 tablet_bootstrap.cc:654] T d99ba4dcbe7f46f38b1bfeb83add08e3 P 2907452cbb30499cbf4c41c42ae48edc: Neither blocks nor log segments found. Creating new log.
I20260812 05:20:21.844909 1472 tablet_bootstrap.cc:492] T d99ba4dcbe7f46f38b1bfeb83add08e3 P 06646753b0e048e187a5cb4fda7858e8: Bootstrap starting.
I20260812 05:20:21.845144 1473 tablet_bootstrap.cc:492] T d99ba4dcbe7f46f38b1bfeb83add08e3 P 4e067bcd6df94544b2d2a9f06efb3948: Bootstrap starting.
I20260812 05:20:21.845589 1472 tablet_bootstrap.cc:654] T d99ba4dcbe7f46f38b1bfeb83add08e3 P 06646753b0e048e187a5cb4fda7858e8: Neither blocks nor log segments found. Creating new log.
I20260812 05:20:21.845610 1470 log.cc:824] T d99ba4dcbe7f46f38b1bfeb83add08e3 P 2907452cbb30499cbf4c41c42ae48edc: Log is configured to *not* fsync() on all Append() calls
I20260812 05:20:21.845813 1473 tablet_bootstrap.cc:654] T d99ba4dcbe7f46f38b1bfeb83add08e3 P 4e067bcd6df94544b2d2a9f06efb3948: Neither blocks nor log segments found. Creating new log.
I20260812 05:20:21.846248 1472 log.cc:824] T d99ba4dcbe7f46f38b1bfeb83add08e3 P 06646753b0e048e187a5cb4fda7858e8: Log is configured to *not* fsync() on all Append() calls
I20260812 05:20:21.846347 1473 log.cc:824] T d99ba4dcbe7f46f38b1bfeb83add08e3 P 4e067bcd6df94544b2d2a9f06efb3948: Log is configured to *not* fsync() on all Append() calls
I20260812 05:20:21.846487 1470 tablet_bootstrap.cc:492] T d99ba4dcbe7f46f38b1bfeb83add08e3 P 2907452cbb30499cbf4c41c42ae48edc: No bootstrap required, opened a new log
I20260812 05:20:21.846553 1470 ts_tablet_manager.cc:1397] T d99ba4dcbe7f46f38b1bfeb83add08e3 P 2907452cbb30499cbf4c41c42ae48edc: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260812 05:20:21.846989 1473 tablet_bootstrap.cc:492] T d99ba4dcbe7f46f38b1bfeb83add08e3 P 4e067bcd6df94544b2d2a9f06efb3948: No bootstrap required, opened a new log
I20260812 05:20:21.847046 1473 ts_tablet_manager.cc:1397] T d99ba4dcbe7f46f38b1bfeb83add08e3 P 4e067bcd6df94544b2d2a9f06efb3948: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260812 05:20:21.847082 1472 tablet_bootstrap.cc:492] T d99ba4dcbe7f46f38b1bfeb83add08e3 P 06646753b0e048e187a5cb4fda7858e8: No bootstrap required, opened a new log
I20260812 05:20:21.847148 1472 ts_tablet_manager.cc:1397] T d99ba4dcbe7f46f38b1bfeb83add08e3 P 06646753b0e048e187a5cb4fda7858e8: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260812 05:20:21.848466 1470 raft_consensus.cc:348] T d99ba4dcbe7f46f38b1bfeb83add08e3 P 2907452cbb30499cbf4c41c42ae48edc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2907452cbb30499cbf4c41c42ae48edc" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 38737 } } peers { permanent_uuid: "06646753b0e048e187a5cb4fda7858e8" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 41909 } } peers { permanent_uuid: "4e067bcd6df94544b2d2a9f06efb3948" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 33241 } }
I20260812 05:20:21.848657 1470 raft_consensus.cc:374] T d99ba4dcbe7f46f38b1bfeb83add08e3 P 2907452cbb30499cbf4c41c42ae48edc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 05:20:21.848694 1470 raft_consensus.cc:729] T d99ba4dcbe7f46f38b1bfeb83add08e3 P 2907452cbb30499cbf4c41c42ae48edc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2907452cbb30499cbf4c41c42ae48edc, State: Initialized, Role: FOLLOWER
I20260812 05:20:21.848676 1473 raft_consensus.cc:348] T d99ba4dcbe7f46f38b1bfeb83add08e3 P 4e067bcd6df94544b2d2a9f06efb3948 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2907452cbb30499cbf4c41c42ae48edc" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 38737 } } peers { permanent_uuid: "06646753b0e048e187a5cb4fda7858e8" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 41909 } } peers { permanent_uuid: "4e067bcd6df94544b2d2a9f06efb3948" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 33241 } }
I20260812 05:20:21.848829 1470 consensus_queue.cc:260] T d99ba4dcbe7f46f38b1bfeb83add08e3 P 2907452cbb30499cbf4c41c42ae48edc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2907452cbb30499cbf4c41c42ae48edc" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 38737 } } peers { permanent_uuid: "06646753b0e048e187a5cb4fda7858e8" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 41909 } } peers { permanent_uuid: "4e067bcd6df94544b2d2a9f06efb3948" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 33241 } }
I20260812 05:20:21.848850 1473 raft_consensus.cc:374] T d99ba4dcbe7f46f38b1bfeb83add08e3 P 4e067bcd6df94544b2d2a9f06efb3948 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 05:20:21.848904 1473 raft_consensus.cc:729] T d99ba4dcbe7f46f38b1bfeb83add08e3 P 4e067bcd6df94544b2d2a9f06efb3948 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e067bcd6df94544b2d2a9f06efb3948, State: Initialized, Role: FOLLOWER
I20260812 05:20:21.849014 1473 consensus_queue.cc:260] T d99ba4dcbe7f46f38b1bfeb83add08e3 P 4e067bcd6df94544b2d2a9f06efb3948 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2907452cbb30499cbf4c41c42ae48edc" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 38737 } } peers { permanent_uuid: "06646753b0e048e187a5cb4fda7858e8" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 41909 } } peers { permanent_uuid: "4e067bcd6df94544b2d2a9f06efb3948" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 33241 } }
I20260812 05:20:21.849108 1470 ts_tablet_manager.cc:1428] T d99ba4dcbe7f46f38b1bfeb83add08e3 P 2907452cbb30499cbf4c41c42ae48edc: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260812 05:20:21.849098 1472 raft_consensus.cc:348] T d99ba4dcbe7f46f38b1bfeb83add08e3 P 06646753b0e048e187a5cb4fda7858e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2907452cbb30499cbf4c41c42ae48edc" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 38737 } } peers { permanent_uuid: "06646753b0e048e187a5cb4fda7858e8" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 41909 } } peers { permanent_uuid: "4e067bcd6df94544b2d2a9f06efb3948" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 33241 } }
I20260812 05:20:21.849220 1222 heartbeater.cc:503] Master 127.31.100.62:33635 was elected leader, sending a full tablet report...
I20260812 05:20:21.849252 1472 raft_consensus.cc:374] T d99ba4dcbe7f46f38b1bfeb83add08e3 P 06646753b0e048e187a5cb4fda7858e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 05:20:21.849283 1473 ts_tablet_manager.cc:1428] T d99ba4dcbe7f46f38b1bfeb83add08e3 P 4e067bcd6df94544b2d2a9f06efb3948: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260812 05:20:21.849288 1472 raft_consensus.cc:729] T d99ba4dcbe7f46f38b1bfeb83add08e3 P 06646753b0e048e187a5cb4fda7858e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06646753b0e048e187a5cb4fda7858e8, State: Initialized, Role: FOLLOWER
I20260812 05:20:21.849385 1006 heartbeater.cc:503] Master 127.31.100.62:33635 was elected leader, sending a full tablet report...
I20260812 05:20:21.849413 1472 consensus_queue.cc:260] T d99ba4dcbe7f46f38b1bfeb83add08e3 P 06646753b0e048e187a5cb4fda7858e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2907452cbb30499cbf4c41c42ae48edc" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 38737 } } peers { permanent_uuid: "06646753b0e048e187a5cb4fda7858e8" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 41909 } } peers { permanent_uuid: "4e067bcd6df94544b2d2a9f06efb3948" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 33241 } }
I20260812 05:20:21.849690 1472 ts_tablet_manager.cc:1428] T d99ba4dcbe7f46f38b1bfeb83add08e3 P 06646753b0e048e187a5cb4fda7858e8: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260812 05:20:21.849799 1452 heartbeater.cc:503] Master 127.31.100.62:33635 was elected leader, sending a full tablet report...
I20260812 05:20:21.854274 1481 raft_consensus.cc:482] T d99ba4dcbe7f46f38b1bfeb83add08e3 P 4e067bcd6df94544b2d2a9f06efb3948 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 05:20:21.854339 1481 raft_consensus.cc:504] T d99ba4dcbe7f46f38b1bfeb83add08e3 P 4e067bcd6df94544b2d2a9f06efb3948 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2907452cbb30499cbf4c41c42ae48edc" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 38737 } } peers { permanent_uuid: "06646753b0e048e187a5cb4fda7858e8" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 41909 } } peers { permanent_uuid: "4e067bcd6df94544b2d2a9f06efb3948" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 33241 } }
I20260812 05:20:21.854619 1481 leader_election.cc:288] T d99ba4dcbe7f46f38b1bfeb83add08e3 P 4e067bcd6df94544b2d2a9f06efb3948 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2907452cbb30499cbf4c41c42ae48edc (127.31.100.2:38737), 06646753b0e048e187a5cb4fda7858e8 (127.31.100.3:41909)
I20260812 05:20:21.858094 1468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:21.854811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.100.1:34219 (local address 127.31.100.2:38737)
0812 05:20:21.854887 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0812 05:20:21.854889 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 05:20:21.854898 (+ 9us) server_negotiation.cc:407] Connection header received
0812 05:20:21.854918 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:20:21.854919 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:20:21.854928 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:20:21.854982 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 05:20:21.855116 (+ 134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:21.855812 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:21.857125 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:21.857672 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:21.857745 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:21.857753 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0812 05:20:21.857998 (+ 245us) server_negotiation.cc:299] Negotiation successful
0812 05:20:21.858036 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260812 05:20:21.858103 1466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:21.854892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.100.1:45151 (local address 127.31.100.3:41909)
0812 05:20:21.854927 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0812 05:20:21.854929 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 05:20:21.855027 (+ 98us) server_negotiation.cc:407] Connection header received
0812 05:20:21.855081 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:20:21.855082 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:20:21.855090 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:20:21.855136 (+ 46us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 05:20:21.855247 (+ 111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:21.855912 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:21.857253 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:21.857812 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:21.857861 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:21.857866 (+ 5us) server_negotiation.cc:1035] Waiting for connection context
0812 05:20:21.858022 (+ 156us) server_negotiation.cc:299] Negotiation successful
0812 05:20:21.858052 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260812 05:20:21.858114 1010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:21.854753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.2:38737 (local address 127.31.100.1:34219)
0812 05:20:21.854801 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0812 05:20:21.854811 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0812 05:20:21.854846 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:20:21.854989 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:20:21.854989 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:20:21.854992 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 05:20:21.855052 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:20:21.855053 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:20:21.855888 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:20:21.855889 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:20:21.857039 (+ 1150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:20:21.857042 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:20:21.857821 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:20:21.857823 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:20:21.857908 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:21.857912 (+ 4us) client_negotiation.cc:769] Sending connection context
0812 05:20:21.857991 (+ 79us) client_negotiation.cc:240] Negotiation successful
0812 05:20:21.858037 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32,"mutex_wait_us":2}
I20260812 05:20:21.858121 1484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:21.854836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.3:41909 (local address 127.31.100.1:45151)
0812 05:20:21.854972 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0812 05:20:21.854981 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0812 05:20:21.855033 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:20:21.855146 (+ 113us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:20:21.855147 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:20:21.855148 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 05:20:21.855200 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:20:21.855201 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:20:21.855971 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:20:21.855972 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:20:21.857186 (+ 1214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:20:21.857188 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:20:21.857885 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:20:21.857886 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:20:21.857956 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:21.857959 (+ 3us) client_negotiation.cc:769] Sending connection context
0812 05:20:21.858005 (+ 46us) client_negotiation.cc:240] Negotiation successful
0812 05:20:21.858047 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"mutex_wait_us":11,"thread_start_us":94,"threads_started":1}
I20260812 05:20:21.858423 1145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d99ba4dcbe7f46f38b1bfeb83add08e3" candidate_uuid: "4e067bcd6df94544b2d2a9f06efb3948" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2907452cbb30499cbf4c41c42ae48edc" is_pre_election: true
I20260812 05:20:21.858585 1145 raft_consensus.cc:2454] T d99ba4dcbe7f46f38b1bfeb83add08e3 P 2907452cbb30499cbf4c41c42ae48edc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e067bcd6df94544b2d2a9f06efb3948 in term 0.
I20260812 05:20:21.858613 1377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d99ba4dcbe7f46f38b1bfeb83add08e3" candidate_uuid: "4e067bcd6df94544b2d2a9f06efb3948" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06646753b0e048e187a5cb4fda7858e8" is_pre_election: true
I20260812 05:20:21.858747 1377 raft_consensus.cc:2454] T d99ba4dcbe7f46f38b1bfeb83add08e3 P 06646753b0e048e187a5cb4fda7858e8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e067bcd6df94544b2d2a9f06efb3948 in term 0.
I20260812 05:20:21.858937 813 leader_election.cc:302] T d99ba4dcbe7f46f38b1bfeb83add08e3 P 4e067bcd6df94544b2d2a9f06efb3948 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06646753b0e048e187a5cb4fda7858e8, 4e067bcd6df94544b2d2a9f06efb3948; no voters:
I20260812 05:20:21.859105 1481 raft_consensus.cc:2781] T d99ba4dcbe7f46f38b1bfeb83add08e3 P 4e067bcd6df94544b2d2a9f06efb3948 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 05:20:21.859144 1481 raft_consensus.cc:482] T d99ba4dcbe7f46f38b1bfeb83add08e3 P 4e067bcd6df94544b2d2a9f06efb3948 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 05:20:21.859158 1481 raft_consensus.cc:3037] T d99ba4dcbe7f46f38b1bfeb83add08e3 P 4e067bcd6df94544b2d2a9f06efb3948 [term 0 FOLLOWER]: Advancing to term 1
I20260812 05:20:21.859835 1481 raft_consensus.cc:504] T d99ba4dcbe7f46f38b1bfeb83add08e3 P 4e067bcd6df94544b2d2a9f06efb3948 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2907452cbb30499cbf4c41c42ae48edc" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 38737 } } peers { permanent_uuid: "06646753b0e048e187a5cb4fda7858e8" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 41909 } } peers { permanent_uuid: "4e067bcd6df94544b2d2a9f06efb3948" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 33241 } }
I20260812 05:20:21.859942 1481 leader_election.cc:288] T d99ba4dcbe7f46f38b1bfeb83add08e3 P 4e067bcd6df94544b2d2a9f06efb3948 [CANDIDATE]: Term 1 election: Requested vote from peers 2907452cbb30499cbf4c41c42ae48edc (127.31.100.2:38737), 06646753b0e048e187a5cb4fda7858e8 (127.31.100.3:41909)
I20260812 05:20:21.860152 1145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d99ba4dcbe7f46f38b1bfeb83add08e3" candidate_uuid: "4e067bcd6df94544b2d2a9f06efb3948" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2907452cbb30499cbf4c41c42ae48edc"
I20260812 05:20:21.860160 1377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d99ba4dcbe7f46f38b1bfeb83add08e3" candidate_uuid: "4e067bcd6df94544b2d2a9f06efb3948" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06646753b0e048e187a5cb4fda7858e8"
I20260812 05:20:21.860221 1145 raft_consensus.cc:3037] T d99ba4dcbe7f46f38b1bfeb83add08e3 P 2907452cbb30499cbf4c41c42ae48edc [term 0 FOLLOWER]: Advancing to term 1
I20260812 05:20:21.860221 1377 raft_consensus.cc:3037] T d99ba4dcbe7f46f38b1bfeb83add08e3 P 06646753b0e048e187a5cb4fda7858e8 [term 0 FOLLOWER]: Advancing to term 1
I20260812 05:20:21.860998 1377 raft_consensus.cc:2454] T d99ba4dcbe7f46f38b1bfeb83add08e3 P 06646753b0e048e187a5cb4fda7858e8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e067bcd6df94544b2d2a9f06efb3948 in term 1.
I20260812 05:20:21.861086 1145 raft_consensus.cc:2454] T d99ba4dcbe7f46f38b1bfeb83add08e3 P 2907452cbb30499cbf4c41c42ae48edc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e067bcd6df94544b2d2a9f06efb3948 in term 1.
I20260812 05:20:21.861159 813 leader_election.cc:302] T d99ba4dcbe7f46f38b1bfeb83add08e3 P 4e067bcd6df94544b2d2a9f06efb3948 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06646753b0e048e187a5cb4fda7858e8, 4e067bcd6df94544b2d2a9f06efb3948; no voters:
I20260812 05:20:21.861263 1481 raft_consensus.cc:2781] T d99ba4dcbe7f46f38b1bfeb83add08e3 P 4e067bcd6df94544b2d2a9f06efb3948 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 05:20:21.861406 1481 raft_consensus.cc:686] T d99ba4dcbe7f46f38b1bfeb83add08e3 P 4e067bcd6df94544b2d2a9f06efb3948 [term 1 LEADER]: Becoming Leader. State: Replica: 4e067bcd6df94544b2d2a9f06efb3948, State: Running, Role: LEADER
I20260812 05:20:21.861507 1481 consensus_queue.cc:237] T d99ba4dcbe7f46f38b1bfeb83add08e3 P 4e067bcd6df94544b2d2a9f06efb3948 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2907452cbb30499cbf4c41c42ae48edc" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 38737 } } peers { permanent_uuid: "06646753b0e048e187a5cb4fda7858e8" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 41909 } } peers { permanent_uuid: "4e067bcd6df94544b2d2a9f06efb3948" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 33241 } }
I20260812 05:20:21.862244 675 catalog_manager.cc:5495] T d99ba4dcbe7f46f38b1bfeb83add08e3 P 4e067bcd6df94544b2d2a9f06efb3948 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4e067bcd6df94544b2d2a9f06efb3948 (127.31.100.1). New cstate: current_term: 1 leader_uuid: "4e067bcd6df94544b2d2a9f06efb3948" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2907452cbb30499cbf4c41c42ae48edc" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 38737 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "06646753b0e048e187a5cb4fda7858e8" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 41909 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4e067bcd6df94544b2d2a9f06efb3948" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 33241 } health_report { overall_health: HEALTHY } } }
Aug 12 05:20:21 dist-test-slave-42z9 krb5kdc[594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512021, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.100.1@KRBTEST.COM
I20260812 05:20:21.871054 1469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:21.864362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57890 (local address 127.31.100.1:33241)
0812 05:20:21.864410 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0812 05:20:21.864411 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0812 05:20:21.864542 (+ 131us) server_negotiation.cc:407] Connection header received
0812 05:20:21.864598 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:20:21.864599 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:20:21.864608 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:20:21.864658 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:20:21.865177 (+ 519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:21.865660 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:21.866338 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:21.866520 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:21.866572 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:21.868678 (+ 2106us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 05:20:21.868685 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 05:20:21.868685 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 05:20:21.868692 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 05:20:21.870185 (+ 1493us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:20:21.870481 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:20:21.870483 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:20:21.870484 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:20:21.870534 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:20:21.870664 (+ 130us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:20:21.870664 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:20:21.870665 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:20:21.870773 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 05:20:21.870893 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0812 05:20:21.870925 (+ 32us) server_negotiation.cc:299] Negotiation successful
0812 05:20:21.870995 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
Aug 12 05:20:21 dist-test-slave-42z9 krb5kdc[594](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512021, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 12 05:20:21 dist-test-slave-42z9 krb5kdc[594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512021, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.31.100.1@KRBTEST.COM
I20260812 05:20:21.893496 1469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:21.887174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57902 (local address 127.31.100.1:33241)
0812 05:20:21.887223 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0812 05:20:21.887225 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 05:20:21.887288 (+ 63us) server_negotiation.cc:407] Connection header received
0812 05:20:21.887355 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:20:21.887356 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:20:21.887365 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:20:21.887412 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:20:21.887941 (+ 529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:21.888484 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:21.889176 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:21.889372 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:21.889430 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:21.891426 (+ 1996us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 05:20:21.891428 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 05:20:21.891428 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 05:20:21.891435 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 05:20:21.892456 (+ 1021us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:20:21.892742 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:20:21.892743 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:20:21.892744 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:20:21.892803 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:20:21.892969 (+ 166us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:20:21.892970 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:20:21.892970 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:20:21.893061 (+ 91us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 05:20:21.893147 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0812 05:20:21.893301 (+ 154us) server_negotiation.cc:299] Negotiation successful
0812 05:20:21.893375 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
Aug 12 05:20:21 dist-test-slave-42z9 krb5kdc[594](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512021, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
W20260812 05:20:21.911933 1223 tablet.cc:2369] T d99ba4dcbe7f46f38b1bfeb83add08e3 P 2907452cbb30499cbf4c41c42ae48edc: Can't schedule compaction. Clean time has not been advanced past its initial value.
Aug 12 05:20:21 dist-test-slave-42z9 krb5kdc[594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512021, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.100.1@KRBTEST.COM
I20260812 05:20:21.914675 1469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:21.908200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57912 (local address 127.31.100.1:33241)
0812 05:20:21.908253 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0812 05:20:21.908255 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 05:20:21.908317 (+ 62us) server_negotiation.cc:407] Connection header received
0812 05:20:21.908407 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:20:21.908408 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:20:21.908418 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:20:21.908486 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:20:21.909065 (+ 579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:21.909587 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:21.910377 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:21.910588 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:21.910638 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:21.912637 (+ 1999us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 05:20:21.912638 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 05:20:21.912638 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 05:20:21.912645 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 05:20:21.913883 (+ 1238us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:20:21.914167 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:20:21.914168 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:20:21.914168 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:20:21.914214 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:20:21.914328 (+ 114us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:20:21.914329 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:20:21.914329 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:20:21.914402 (+ 73us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 05:20:21.914486 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0812 05:20:21.914557 (+ 71us) server_negotiation.cc:299] Negotiation successful
0812 05:20:21.914613 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
W20260812 05:20:21.915092 852 tablet_service.cc:2908] Rejecting scan request for tablet d99ba4dcbe7f46f38b1bfeb83add08e3: Uninitialized: clean time has not yet been initialized
I20260812 05:20:21.915829 32144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0waFvT/build/release/bin/kudu with pid 780
I20260812 05:20:21.927834 32144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0waFvT/build/release/bin/kudu with pid 1015
I20260812 05:20:21.939489 32144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0waFvT/build/release/bin/kudu with pid 1229
I20260812 05:20:21.948239 32144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0waFvT/build/release/bin/kudu with pid 618
2026-08-12T05:20:21Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2053 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786512017613366-32144-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 05:20:22 dist-test-slave-42z9 krb5kdc[1515](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 05:20:22 dist-test-slave-42z9 krb5kdc[1515](info): set up 2 sockets
Aug 12 05:20:22 dist-test-slave-42z9 krb5kdc[1515](info): commencing operation
krb5kdc: starting...
W20260812 05:20:23.116050 32144 mini_kdc.cc:121] Time spent starting KDC: real 1.143s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786512017613366-32144-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786512017613366-32144-0/krb5kdc/test-user.keytab.
Aug 12 05:20:23 dist-test-slave-42z9 krb5kdc[1515](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512023, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T05:20:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T05:20:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.100.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.100.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.100.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.62.keytab.
Entry for principal kudu/127.31.100.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.62.keytab.
I20260812 05:20:23.228482 32144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0waFvT/build/release/bin/kudu
/tmp/dist-test-task0waFvT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786512017613366-32144-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786512017613366-32144-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786512017613366-32144-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786512017613366-32144-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.100.62:35317
--webserver_interface=127.31.100.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.62.keytab
--principal=kudu/127.31.100.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.100.20:39789
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.100.62:35317
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786512017613366-32144-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786512017613366-32144-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786512017613366-32144-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 05:20:23.310376 1538 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 05:20:23.310586 1538 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 05:20:23.310616 1538 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 05:20:23.311757 1538 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 05:20:23.311797 1538 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260812 05:20:23.311815 1538 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 05:20:23.311829 1538 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 05:20:23.311844 1538 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 05:20:23.311862 1538 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 05:20:23.313499 1538 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.100.20:39789
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786512017613366-32144-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786512017613366-32144-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.100.62:35317
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.62.keytab
--principal=kudu/127.31.100.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.100.62:35317
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786512017613366-32144-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.100.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1538
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786512017613366-32144-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 04:20:34 UTC on ccycloud.quasar-phqdqp.root.comops.site
build id 4581871
I20260812 05:20:23.313740 1538 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 05:20:23.313951 1538 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 05:20:23.318516 1551 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 05:20:23.318550 1556 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 05:20:23.318544 1552 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 05:20:23.318835 1538 server_base.cc:1034] running on GCE node
I20260812 05:20:23.319072 1538 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 05:20:23.319346 1538 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 05:20:23.320500 1538 hybrid_clock.cc:648] HybridClock initialized: now 1786512023320501 us; error 44 us; skew 500 ppm
Aug 12 05:20:23 dist-test-slave-42z9 krb5kdc[1515](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512023, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.100.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 05:20:23.327399 1538 init.cc:377] Logged in from keytab as kudu/127.31.100.62@KRBTEST.COM (short username kudu)
I20260812 05:20:23.327795 1538 webserver.cc:460] Webserver started at http://127.31.100.62:33271/ using document root <none> and password file <none>
I20260812 05:20:23.328017 1538 fs_manager.cc:359] Metadata directory not provided
I20260812 05:20:23.328065 1538 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 05:20:23.328193 1538 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 05:20:23.329217 1538 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786512017613366-32144-0/minicluster-data/master-0/data/instance:
uuid: "3d208a9824f844afb0da31ac8bb9bc3a"
format_stamp: "Formatted at 2026-08-12 05:20:23 on dist-test-slave-42z9"
server_key: "6ad5400ac4f52b06da9d7aaa27109bbd"
server_key_iv: "73f37b5c31482d2ddcc22162bdc72c22"
server_key_version: "encryptionkey@0"
I20260812 05:20:23.329576 1538 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786512017613366-32144-0/minicluster-data/master-0/wal/instance:
uuid: "3d208a9824f844afb0da31ac8bb9bc3a"
format_stamp: "Formatted at 2026-08-12 05:20:23 on dist-test-slave-42z9"
server_key: "6ad5400ac4f52b06da9d7aaa27109bbd"
server_key_iv: "73f37b5c31482d2ddcc22162bdc72c22"
server_key_version: "encryptionkey@0"
I20260812 05:20:23.330850 1538 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260812 05:20:23.331579 1563 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 05:20:23.331732 1538 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260812 05:20:23.331823 1538 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786512017613366-32144-0/minicluster-data/master-0/data,/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786512017613366-32144-0/minicluster-data/master-0/wal
uuid: "3d208a9824f844afb0da31ac8bb9bc3a"
format_stamp: "Formatted at 2026-08-12 05:20:23 on dist-test-slave-42z9"
server_key: "6ad5400ac4f52b06da9d7aaa27109bbd"
server_key_iv: "73f37b5c31482d2ddcc22162bdc72c22"
server_key_version: "encryptionkey@0"
I20260812 05:20:23.331885 1538 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786512017613366-32144-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786512017613366-32144-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786512017613366-32144-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 05:20:23.341670 1538 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 05:20:23.343467 1538 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 05:20:23.343593 1538 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 05:20:23.348244 1538 rpc_server.cc:307] RPC server started. Bound to: 127.31.100.62:35317
I20260812 05:20:23.348308 1668 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.100.62:35317 every 8 connection(s)
I20260812 05:20:23.348589 1538 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786512017613366-32144-0/minicluster-data/master-0/data/info.pb
I20260812 05:20:23.349247 1669 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 05:20:23.351522 1669 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3d208a9824f844afb0da31ac8bb9bc3a: Bootstrap starting.
I20260812 05:20:23.352057 1669 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3d208a9824f844afb0da31ac8bb9bc3a: Neither blocks nor log segments found. Creating new log.
I20260812 05:20:23.352289 1669 log.cc:824] T 00000000000000000000000000000000 P 3d208a9824f844afb0da31ac8bb9bc3a: Log is configured to *not* fsync() on all Append() calls
I20260812 05:20:23.352921 1669 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3d208a9824f844afb0da31ac8bb9bc3a: No bootstrap required, opened a new log
I20260812 05:20:23.353154 32144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0waFvT/build/release/bin/kudu as pid 1538
I20260812 05:20:23.353245 32144 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786512017613366-32144-0/minicluster-data/master-0/wal/instance
I20260812 05:20:23.353389 32144 external_mini_cluster.cc:1428] Setting key 40ff6a20eedf012cf0b750800d3ab197
I20260812 05:20:23.355403 1669 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3d208a9824f844afb0da31ac8bb9bc3a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d208a9824f844afb0da31ac8bb9bc3a" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 35317 } }
I20260812 05:20:23.355590 1669 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3d208a9824f844afb0da31ac8bb9bc3a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 05:20:23.355626 1669 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3d208a9824f844afb0da31ac8bb9bc3a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d208a9824f844afb0da31ac8bb9bc3a, State: Initialized, Role: FOLLOWER
I20260812 05:20:23.355749 1669 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3d208a9824f844afb0da31ac8bb9bc3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d208a9824f844afb0da31ac8bb9bc3a" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 35317 } }
I20260812 05:20:23.355834 1669 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3d208a9824f844afb0da31ac8bb9bc3a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 05:20:23.355873 1669 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3d208a9824f844afb0da31ac8bb9bc3a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 05:20:23.355923 1669 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3d208a9824f844afb0da31ac8bb9bc3a [term 0 FOLLOWER]: Advancing to term 1
I20260812 05:20:23.356418 1669 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3d208a9824f844afb0da31ac8bb9bc3a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d208a9824f844afb0da31ac8bb9bc3a" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 35317 } }
I20260812 05:20:23.356526 1669 leader_election.cc:302] T 00000000000000000000000000000000 P 3d208a9824f844afb0da31ac8bb9bc3a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3d208a9824f844afb0da31ac8bb9bc3a; no voters:
I20260812 05:20:23.356681 1669 leader_election.cc:288] T 00000000000000000000000000000000 P 3d208a9824f844afb0da31ac8bb9bc3a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 05:20:23.356721 1677 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3d208a9824f844afb0da31ac8bb9bc3a [term 1 FOLLOWER]: Leader election won for term 1
I20260812 05:20:23.356839 1677 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3d208a9824f844afb0da31ac8bb9bc3a [term 1 LEADER]: Becoming Leader. State: Replica: 3d208a9824f844afb0da31ac8bb9bc3a, State: Running, Role: LEADER
I20260812 05:20:23.356931 1669 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3d208a9824f844afb0da31ac8bb9bc3a [sys.catalog]: configured and running, proceeding with master startup.
I20260812 05:20:23.356958 1677 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3d208a9824f844afb0da31ac8bb9bc3a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d208a9824f844afb0da31ac8bb9bc3a" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 35317 } }
I20260812 05:20:23.357249 1677 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d208a9824f844afb0da31ac8bb9bc3a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3d208a9824f844afb0da31ac8bb9bc3a. Latest consensus state: current_term: 1 leader_uuid: "3d208a9824f844afb0da31ac8bb9bc3a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d208a9824f844afb0da31ac8bb9bc3a" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 35317 } } }
I20260812 05:20:23.357318 1677 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d208a9824f844afb0da31ac8bb9bc3a [sys.catalog]: This master's current role is: LEADER
I20260812 05:20:23.358368 1681 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d208a9824f844afb0da31ac8bb9bc3a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3d208a9824f844afb0da31ac8bb9bc3a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d208a9824f844afb0da31ac8bb9bc3a" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 35317 } } }
W20260812 05:20:23.358681 1705 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3d208a9824f844afb0da31ac8bb9bc3a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260812 05:20:23.359236 1681 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d208a9824f844afb0da31ac8bb9bc3a [sys.catalog]: This master's current role is: LEADER
W20260812 05:20:23.359246 1705 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 05:20:23.359328 1687 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 05:20:23.359556 1687 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Aug 12 05:20:23 dist-test-slave-42z9 krb5kdc[1515](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512023, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.100.62@KRBTEST.COM
I20260812 05:20:23.361163 1687 catalog_manager.cc:1347] Generated new cluster ID: 4e3e901392644cdfab4e9b05c2c347fd
I20260812 05:20:23.361212 1687 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 05:20:23.365355 1676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:23.353852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34590 (local address 127.31.100.62:35317)
0812 05:20:23.354023 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0812 05:20:23.354031 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 05:20:23.354056 (+ 25us) server_negotiation.cc:407] Connection header received
0812 05:20:23.354995 (+ 939us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:20:23.355003 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:20:23.355057 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:20:23.355167 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:20:23.355948 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:23.357043 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:23.358034 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:23.358596 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:23.358711 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:23.361296 (+ 2585us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 05:20:23.361300 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 05:20:23.361300 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 05:20:23.361312 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0812 05:20:23.363803 (+ 2491us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:20:23.364180 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:20:23.364183 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:20:23.364184 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:20:23.364310 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:20:23.364490 (+ 180us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:20:23.364491 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:20:23.364491 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:20:23.364789 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 05:20:23.364955 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0812 05:20:23.365023 (+ 68us) server_negotiation.cc:299] Negotiation successful
0812 05:20:23.365158 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":85,"threads_started":1}
I20260812 05:20:23.379985 1687 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 05:20:23.380672 1687 catalog_manager.cc:1504] Loading token signing keys...
I20260812 05:20:23.387199 1687 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3d208a9824f844afb0da31ac8bb9bc3a: Generated new TSK 0
I20260812 05:20:23.387370 1687 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.31.100.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.100.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.100.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.1.keytab.
Entry for principal kudu/127.31.100.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.1.keytab.
I20260812 05:20:23.415659 32144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0waFvT/build/release/bin/kudu
/tmp/dist-test-task0waFvT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786512017613366-32144-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786512017613366-32144-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786512017613366-32144-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786512017613366-32144-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.100.1:0
--local_ip_for_outbound_sockets=127.31.100.1
--webserver_interface=127.31.100.1
--webserver_port=0
--tserver_master_addrs=127.31.100.62:35317
--keytab_file=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.1.keytab
--principal=kudu/127.31.100.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.100.20:39789
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786512017613366-32144-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786512017613366-32144-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786512017613366-32144-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 05:20:23.494585 1712 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 05:20:23.494786 1712 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 05:20:23.494809 1712 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 05:20:23.495913 1712 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 05:20:23.495957 1712 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 05:20:23.495980 1712 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260812 05:20:23.496011 1712 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.100.1
I20260812 05:20:23.497390 1712 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.100.20:39789
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786512017613366-32144-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786512017613366-32144-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.1.keytab
--principal=kudu/127.31.100.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.100.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786512017613366-32144-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.100.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.31.100.62:35317
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1712
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.100.1
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786512017613366-32144-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 04:20:34 UTC on ccycloud.quasar-phqdqp.root.comops.site
build id 4581871
I20260812 05:20:23.497612 1712 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 05:20:23.497812 1712 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 05:20:23.502489 1724 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 05:20:23.502596 1720 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 05:20:23.502501 1722 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 05:20:23.502718 1712 server_base.cc:1034] running on GCE node
I20260812 05:20:23.502993 1712 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 05:20:23.503216 1712 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 05:20:23.504354 1712 hybrid_clock.cc:648] HybridClock initialized: now 1786512023504332 us; error 38 us; skew 500 ppm
Aug 12 05:20:23 dist-test-slave-42z9 krb5kdc[1515](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512023, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.100.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 05:20:23.506803 1712 init.cc:377] Logged in from keytab as kudu/127.31.100.1@KRBTEST.COM (short username kudu)
I20260812 05:20:23.507227 1712 webserver.cc:460] Webserver started at http://127.31.100.1:39577/ using document root <none> and password file <none>
I20260812 05:20:23.507427 1712 fs_manager.cc:359] Metadata directory not provided
I20260812 05:20:23.507478 1712 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 05:20:23.507599 1712 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 05:20:23.508548 1712 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786512017613366-32144-0/minicluster-data/ts-0/data/instance:
uuid: "770f563b32324129ae355bd292cfce2e"
format_stamp: "Formatted at 2026-08-12 05:20:23 on dist-test-slave-42z9"
server_key: "74044e63a75f11fd203910322c36262d"
server_key_iv: "f8d6756f6b3933a76fd43c682e701ee3"
server_key_version: "encryptionkey@0"
I20260812 05:20:23.508855 1712 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786512017613366-32144-0/minicluster-data/ts-0/wal/instance:
uuid: "770f563b32324129ae355bd292cfce2e"
format_stamp: "Formatted at 2026-08-12 05:20:23 on dist-test-slave-42z9"
server_key: "74044e63a75f11fd203910322c36262d"
server_key_iv: "f8d6756f6b3933a76fd43c682e701ee3"
server_key_version: "encryptionkey@0"
I20260812 05:20:23.510126 1712 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260812 05:20:23.510767 1736 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 05:20:23.510929 1712 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260812 05:20:23.511013 1712 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786512017613366-32144-0/minicluster-data/ts-0/data,/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786512017613366-32144-0/minicluster-data/ts-0/wal
uuid: "770f563b32324129ae355bd292cfce2e"
format_stamp: "Formatted at 2026-08-12 05:20:23 on dist-test-slave-42z9"
server_key: "74044e63a75f11fd203910322c36262d"
server_key_iv: "f8d6756f6b3933a76fd43c682e701ee3"
server_key_version: "encryptionkey@0"
I20260812 05:20:23.511077 1712 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786512017613366-32144-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786512017613366-32144-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786512017613366-32144-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 05:20:23.527470 1712 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 05:20:23.529120 1712 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 05:20:23.529237 1712 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 05:20:23.529712 1712 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 05:20:23.529750 1712 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 05:20:23.529770 1712 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 05:20:23.529789 1712 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 05:20:23.536164 1712 rpc_server.cc:307] RPC server started. Bound to: 127.31.100.1:42473
I20260812 05:20:23.536283 1941 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.100.1:42473 every 8 connection(s)
Aug 12 05:20:23 dist-test-slave-42z9 krb5kdc[1515](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512023, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.100.1@KRBTEST.COM for kudu/127.31.100.62@KRBTEST.COM
I20260812 05:20:23.536554 1712 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786512017613366-32144-0/minicluster-data/ts-0/data/info.pb
I20260812 05:20:23.539301 1676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:23.530247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.100.1:45587 (local address 127.31.100.62:35317)
0812 05:20:23.530295 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0812 05:20:23.530298 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 05:20:23.530540 (+ 242us) server_negotiation.cc:407] Connection header received
0812 05:20:23.531007 (+ 467us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:20:23.531009 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:20:23.531023 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:20:23.531092 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:20:23.531893 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:23.532658 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:23.534129 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:23.534436 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:23.534531 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:23.537007 (+ 2476us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 05:20:23.537009 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 05:20:23.537009 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 05:20:23.537017 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0812 05:20:23.538151 (+ 1134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:20:23.538541 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:20:23.538542 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:20:23.538542 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:20:23.538591 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:20:23.538779 (+ 188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:20:23.538780 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:20:23.538780 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:20:23.538894 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 05:20:23.538954 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0812 05:20:23.539156 (+ 202us) server_negotiation.cc:299] Negotiation successful
0812 05:20:23.539233 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260812 05:20:23.539476 1751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:23.530240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.62:35317 (local address 127.31.100.1:45587)
0812 05:20:23.530389 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0812 05:20:23.530448 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0812 05:20:23.530881 (+ 433us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:20:23.531166 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:20:23.531170 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:20:23.531531 (+ 361us) client_negotiation.cc:189] Negotiated authn=SASL
0812 05:20:23.531790 (+ 259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:20:23.531795 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:20:23.532754 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:20:23.532755 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:20:23.534041 (+ 1286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:20:23.534044 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:20:23.534541 (+ 497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:20:23.534542 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:20:23.534664 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:23.535153 (+ 489us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 05:20:23.535163 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0812 05:20:23.536891 (+ 1728us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 05:20:23.538221 (+ 1330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 05:20:23.538228 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 05:20:23.538229 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 05:20:23.538475 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 05:20:23.538633 (+ 158us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 05:20:23.538634 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 05:20:23.538634 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 05:20:23.538732 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 05:20:23.538943 (+ 211us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 05:20:23.538946 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 05:20:23.539065 (+ 119us) client_negotiation.cc:769] Sending connection context
0812 05:20:23.539138 (+ 73us) client_negotiation.cc:240] Negotiation successful
0812 05:20:23.539258 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":77,"threads_started":1}
I20260812 05:20:23.540071 1944 heartbeater.cc:347] Connected to a master server at 127.31.100.62:35317
I20260812 05:20:23.540089 32144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0waFvT/build/release/bin/kudu as pid 1712
I20260812 05:20:23.540170 32144 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786512017613366-32144-0/minicluster-data/ts-0/wal/instance
I20260812 05:20:23.540163 1944 heartbeater.cc:464] Registering TS with master...
I20260812 05:20:23.540284 32144 external_mini_cluster.cc:1428] Setting key 5e2e64498d753bd70a133a18061c0c07
I20260812 05:20:23.540342 1944 heartbeater.cc:511] Master 127.31.100.62:35317 requested a full tablet report, sending...
I20260812 05:20:23.540827 1585 ts_manager.cc:194] Registered new tserver with Master: 770f563b32324129ae355bd292cfce2e (127.31.100.1:42473)
I20260812 05:20:23.541710 1585 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.100.1@KRBTEST.COM'} at 127.31.100.1:45587
I20260812 05:20:23.546711 1957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:23.541283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.62:35317 (local address 127.31.100.1:57779)
0812 05:20:23.541504 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0812 05:20:23.541515 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0812 05:20:23.541574 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:20:23.541687 (+ 113us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:20:23.541688 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:20:23.541892 (+ 204us) client_negotiation.cc:189] Negotiated authn=SASL
0812 05:20:23.541958 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:20:23.541960 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:20:23.542814 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:20:23.542815 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:20:23.543309 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:20:23.543310 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:20:23.543654 (+ 344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:20:23.543655 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:20:23.543729 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:23.544109 (+ 380us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 05:20:23.544114 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0812 05:20:23.544375 (+ 261us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 05:20:23.545805 (+ 1430us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 05:20:23.545806 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 05:20:23.545806 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 05:20:23.545966 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 05:20:23.546141 (+ 175us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 05:20:23.546142 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 05:20:23.546142 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 05:20:23.546200 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 05:20:23.546402 (+ 202us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 05:20:23.546403 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 05:20:23.546456 (+ 53us) client_negotiation.cc:769] Sending connection context
0812 05:20:23.546514 (+ 58us) client_negotiation.cc:240] Negotiation successful
0812 05:20:23.546620 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":119,"threads_started":1}
I20260812 05:20:23.546710 1676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:23.541317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.100.1:57779 (local address 127.31.100.62:35317)
0812 05:20:23.541364 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0812 05:20:23.541367 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 05:20:23.541576 (+ 209us) server_negotiation.cc:407] Connection header received
0812 05:20:23.541618 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:20:23.541619 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:20:23.541630 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:20:23.541682 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:20:23.542032 (+ 350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:23.542721 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:23.543386 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:23.543593 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:23.543647 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:23.544456 (+ 809us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 05:20:23.544457 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 05:20:23.544457 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 05:20:23.544464 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 05:20:23.545732 (+ 1268us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:20:23.546032 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:20:23.546033 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:20:23.546033 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:20:23.546088 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:20:23.546250 (+ 162us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:20:23.546250 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:20:23.546251 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:20:23.546350 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 05:20:23.546404 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0812 05:20:23.546543 (+ 139us) server_negotiation.cc:299] Negotiation successful
0812 05:20:23.546621 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":23}
WARNING: no policy specified for kudu/127.31.100.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.100.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.100.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.2.keytab.
Entry for principal kudu/127.31.100.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.2.keytab.
I20260812 05:20:23.565150 32144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0waFvT/build/release/bin/kudu
/tmp/dist-test-task0waFvT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786512017613366-32144-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786512017613366-32144-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786512017613366-32144-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786512017613366-32144-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.100.2:0
--local_ip_for_outbound_sockets=127.31.100.2
--webserver_interface=127.31.100.2
--webserver_port=0
--tserver_master_addrs=127.31.100.62:35317
--keytab_file=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.2.keytab
--principal=kudu/127.31.100.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.100.20:39789
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786512017613366-32144-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786512017613366-32144-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786512017613366-32144-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 05:20:23.645044 1966 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 05:20:23.645229 1966 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 05:20:23.645251 1966 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 05:20:23.646396 1966 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 05:20:23.646436 1966 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 05:20:23.646476 1966 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260812 05:20:23.646507 1966 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.100.2
I20260812 05:20:23.647881 1966 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.100.20:39789
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786512017613366-32144-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786512017613366-32144-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.2.keytab
--principal=kudu/127.31.100.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.100.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786512017613366-32144-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.100.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.31.100.62:35317
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1966
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.100.2
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786512017613366-32144-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 04:20:34 UTC on ccycloud.quasar-phqdqp.root.comops.site
build id 4581871
I20260812 05:20:23.648097 1966 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 05:20:23.648284 1966 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 05:20:23.652983 1978 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 05:20:23.652971 1974 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 05:20:23.652973 1973 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 05:20:23.653240 1966 server_base.cc:1034] running on GCE node
I20260812 05:20:23.653561 1966 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 05:20:23.653800 1966 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 05:20:23.654945 1966 hybrid_clock.cc:648] HybridClock initialized: now 1786512023654932 us; error 26 us; skew 500 ppm
Aug 12 05:20:23 dist-test-slave-42z9 krb5kdc[1515](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512023, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.100.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 05:20:23.657419 1966 init.cc:377] Logged in from keytab as kudu/127.31.100.2@KRBTEST.COM (short username kudu)
I20260812 05:20:23.657857 1966 webserver.cc:460] Webserver started at http://127.31.100.2:42835/ using document root <none> and password file <none>
I20260812 05:20:23.658059 1966 fs_manager.cc:359] Metadata directory not provided
I20260812 05:20:23.658131 1966 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 05:20:23.658253 1966 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 05:20:23.659297 1966 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786512017613366-32144-0/minicluster-data/ts-1/data/instance:
uuid: "280276f5683543e0aeeebad99ea84a84"
format_stamp: "Formatted at 2026-08-12 05:20:23 on dist-test-slave-42z9"
server_key: "549b187a32b3a365c0e77971b7c852fb"
server_key_iv: "07f4ca5bc0ede34f499ad1ee60458fe2"
server_key_version: "encryptionkey@0"
I20260812 05:20:23.659631 1966 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786512017613366-32144-0/minicluster-data/ts-1/wal/instance:
uuid: "280276f5683543e0aeeebad99ea84a84"
format_stamp: "Formatted at 2026-08-12 05:20:23 on dist-test-slave-42z9"
server_key: "549b187a32b3a365c0e77971b7c852fb"
server_key_iv: "07f4ca5bc0ede34f499ad1ee60458fe2"
server_key_version: "encryptionkey@0"
I20260812 05:20:23.660877 1966 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260812 05:20:23.661597 1988 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 05:20:23.661737 1966 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260812 05:20:23.661820 1966 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786512017613366-32144-0/minicluster-data/ts-1/data,/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786512017613366-32144-0/minicluster-data/ts-1/wal
uuid: "280276f5683543e0aeeebad99ea84a84"
format_stamp: "Formatted at 2026-08-12 05:20:23 on dist-test-slave-42z9"
server_key: "549b187a32b3a365c0e77971b7c852fb"
server_key_iv: "07f4ca5bc0ede34f499ad1ee60458fe2"
server_key_version: "encryptionkey@0"
I20260812 05:20:23.661883 1966 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786512017613366-32144-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786512017613366-32144-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786512017613366-32144-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 05:20:23.673717 1966 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 05:20:23.675316 1966 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 05:20:23.675474 1966 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 05:20:23.675973 1966 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 05:20:23.676011 1966 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 05:20:23.676038 1966 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 05:20:23.676054 1966 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 12 05:20:23 dist-test-slave-42z9 krb5kdc[1515](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512023, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.100.2@KRBTEST.COM for kudu/127.31.100.62@KRBTEST.COM
I20260812 05:20:23.682799 1966 rpc_server.cc:307] RPC server started. Bound to: 127.31.100.2:35527
I20260812 05:20:23.682874 2172 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.100.2:35527 every 8 connection(s)
I20260812 05:20:23.683148 1966 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786512017613366-32144-0/minicluster-data/ts-1/data/info.pb
I20260812 05:20:23.685168 1676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:23.676361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.100.2:38067 (local address 127.31.100.62:35317)
0812 05:20:23.676429 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0812 05:20:23.676431 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 05:20:23.676637 (+ 206us) server_negotiation.cc:407] Connection header received
0812 05:20:23.677074 (+ 437us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:20:23.677075 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:20:23.677088 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:20:23.677160 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:20:23.677841 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:23.678595 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:23.679645 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:23.679854 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:23.679910 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:23.682808 (+ 2898us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 05:20:23.682809 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 05:20:23.682809 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 05:20:23.682818 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0812 05:20:23.684139 (+ 1321us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:20:23.684495 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:20:23.684495 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:20:23.684496 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:20:23.684544 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:20:23.684719 (+ 175us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:20:23.684719 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:20:23.684720 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:20:23.684810 (+ 90us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 05:20:23.684853 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0812 05:20:23.685049 (+ 196us) server_negotiation.cc:299] Negotiation successful
0812 05:20:23.685106 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260812 05:20:23.685343 2003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:23.676333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.62:35317 (local address 127.31.100.2:38067)
0812 05:20:23.676510 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0812 05:20:23.676547 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0812 05:20:23.676987 (+ 440us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:20:23.677233 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:20:23.677236 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:20:23.677572 (+ 336us) client_negotiation.cc:189] Negotiated authn=SASL
0812 05:20:23.677749 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:20:23.677753 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:20:23.678691 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:20:23.678692 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:20:23.679562 (+ 870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:20:23.679564 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:20:23.679923 (+ 359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:20:23.679924 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:20:23.680038 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:23.680502 (+ 464us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 05:20:23.680522 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0812 05:20:23.682713 (+ 2191us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 05:20:23.684204 (+ 1491us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 05:20:23.684210 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 05:20:23.684211 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 05:20:23.684414 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 05:20:23.684590 (+ 176us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 05:20:23.684590 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 05:20:23.684590 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 05:20:23.684679 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 05:20:23.684852 (+ 173us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 05:20:23.684855 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 05:20:23.684965 (+ 110us) client_negotiation.cc:769] Sending connection context
0812 05:20:23.685039 (+ 74us) client_negotiation.cc:240] Negotiation successful
0812 05:20:23.685150 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":112,"threads_started":1}
I20260812 05:20:23.685809 2173 heartbeater.cc:347] Connected to a master server at 127.31.100.62:35317
I20260812 05:20:23.685892 2173 heartbeater.cc:464] Registering TS with master...
I20260812 05:20:23.686079 2173 heartbeater.cc:511] Master 127.31.100.62:35317 requested a full tablet report, sending...
I20260812 05:20:23.686466 1584 ts_manager.cc:194] Registered new tserver with Master: 280276f5683543e0aeeebad99ea84a84 (127.31.100.2:35527)
I20260812 05:20:23.686861 1584 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.100.2@KRBTEST.COM'} at 127.31.100.2:38067
I20260812 05:20:23.689638 32144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0waFvT/build/release/bin/kudu as pid 1966
I20260812 05:20:23.689728 32144 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786512017613366-32144-0/minicluster-data/ts-1/wal/instance
I20260812 05:20:23.689894 32144 external_mini_cluster.cc:1428] Setting key 7eb132501899894feacd535b9de278d1
I20260812 05:20:23.692065 2185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:23.686772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.62:35317 (local address 127.31.100.2:56473)
0812 05:20:23.686896 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0812 05:20:23.686906 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0812 05:20:23.686968 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:20:23.687075 (+ 107us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:20:23.687075 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:20:23.687267 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0812 05:20:23.687320 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:20:23.687322 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:20:23.687983 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:20:23.687983 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:20:23.688486 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:20:23.688487 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:20:23.688788 (+ 301us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:20:23.688788 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:20:23.688858 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:23.689189 (+ 331us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 05:20:23.689192 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0812 05:20:23.689435 (+ 243us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 05:20:23.691000 (+ 1565us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 05:20:23.691000 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 05:20:23.691000 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 05:20:23.691159 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 05:20:23.691361 (+ 202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 05:20:23.691362 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 05:20:23.691362 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 05:20:23.691425 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 05:20:23.691795 (+ 370us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 05:20:23.691796 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 05:20:23.691840 (+ 44us) client_negotiation.cc:769] Sending connection context
0812 05:20:23.691906 (+ 66us) client_negotiation.cc:240] Negotiation successful
0812 05:20:23.691986 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":68,"threads_started":1}
I20260812 05:20:23.692098 1676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:23.686850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.100.2:56473 (local address 127.31.100.62:35317)
0812 05:20:23.686888 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0812 05:20:23.686890 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 05:20:23.686982 (+ 92us) server_negotiation.cc:407] Connection header received
0812 05:20:23.687007 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:20:23.687008 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:20:23.687017 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:20:23.687068 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:20:23.687390 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:23.687900 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:23.688555 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:23.688733 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:23.688786 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:23.689532 (+ 746us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 05:20:23.689533 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 05:20:23.689533 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 05:20:23.689540 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 05:20:23.690925 (+ 1385us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:20:23.691229 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:20:23.691230 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:20:23.691231 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:20:23.691297 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:20:23.691472 (+ 175us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:20:23.691472 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:20:23.691472 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:20:23.691662 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 05:20:23.691727 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0812 05:20:23.691942 (+ 215us) server_negotiation.cc:299] Negotiation successful
0812 05:20:23.692018 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
WARNING: no policy specified for kudu/127.31.100.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.100.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.100.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.3.keytab.
Entry for principal kudu/127.31.100.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.3.keytab.
I20260812 05:20:23.713209 32144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0waFvT/build/release/bin/kudu
/tmp/dist-test-task0waFvT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786512017613366-32144-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786512017613366-32144-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786512017613366-32144-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786512017613366-32144-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.100.3:0
--local_ip_for_outbound_sockets=127.31.100.3
--webserver_interface=127.31.100.3
--webserver_port=0
--tserver_master_addrs=127.31.100.62:35317
--keytab_file=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.3.keytab
--principal=kudu/127.31.100.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.100.20:39789
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786512017613366-32144-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786512017613366-32144-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786512017613366-32144-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 05:20:23.791945 2191 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 05:20:23.792132 2191 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 05:20:23.792155 2191 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 05:20:23.793257 2191 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 05:20:23.793298 2191 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 05:20:23.793316 2191 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260812 05:20:23.793341 2191 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.100.3
I20260812 05:20:23.794842 2191 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.100.20:39789
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786512017613366-32144-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786512017613366-32144-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.3.keytab
--principal=kudu/127.31.100.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.100.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786512017613366-32144-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.100.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.31.100.62:35317
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2191
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.100.3
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786512017613366-32144-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 04:20:34 UTC on ccycloud.quasar-phqdqp.root.comops.site
build id 4581871
I20260812 05:20:23.795049 2191 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 05:20:23.795269 2191 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 05:20:23.799948 2197 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 05:20:23.799957 2202 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 05:20:23.800034 2198 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 05:20:23.800122 2191 server_base.cc:1034] running on GCE node
I20260812 05:20:23.800352 2191 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 05:20:23.800541 2191 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 05:20:23.801676 2191 hybrid_clock.cc:648] HybridClock initialized: now 1786512023801662 us; error 28 us; skew 500 ppm
Aug 12 05:20:23 dist-test-slave-42z9 krb5kdc[1515](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512023, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.100.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 05:20:23.804018 2191 init.cc:377] Logged in from keytab as kudu/127.31.100.3@KRBTEST.COM (short username kudu)
I20260812 05:20:23.804373 2191 webserver.cc:460] Webserver started at http://127.31.100.3:38951/ using document root <none> and password file <none>
I20260812 05:20:23.804538 2191 fs_manager.cc:359] Metadata directory not provided
I20260812 05:20:23.804576 2191 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 05:20:23.804662 2191 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 05:20:23.805601 2191 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786512017613366-32144-0/minicluster-data/ts-2/data/instance:
uuid: "2e7c1aeb46b547e39cddeae55b4bd2a4"
format_stamp: "Formatted at 2026-08-12 05:20:23 on dist-test-slave-42z9"
server_key: "b12103393d31d6120503441213fee173"
server_key_iv: "24fedd0e25790d4b1c655c047a468317"
server_key_version: "encryptionkey@0"
I20260812 05:20:23.805878 2191 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786512017613366-32144-0/minicluster-data/ts-2/wal/instance:
uuid: "2e7c1aeb46b547e39cddeae55b4bd2a4"
format_stamp: "Formatted at 2026-08-12 05:20:23 on dist-test-slave-42z9"
server_key: "b12103393d31d6120503441213fee173"
server_key_iv: "24fedd0e25790d4b1c655c047a468317"
server_key_version: "encryptionkey@0"
I20260812 05:20:23.807058 2191 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260812 05:20:23.807720 2211 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 05:20:23.807861 2191 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260812 05:20:23.807950 2191 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786512017613366-32144-0/minicluster-data/ts-2/data,/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786512017613366-32144-0/minicluster-data/ts-2/wal
uuid: "2e7c1aeb46b547e39cddeae55b4bd2a4"
format_stamp: "Formatted at 2026-08-12 05:20:23 on dist-test-slave-42z9"
server_key: "b12103393d31d6120503441213fee173"
server_key_iv: "24fedd0e25790d4b1c655c047a468317"
server_key_version: "encryptionkey@0"
I20260812 05:20:23.808007 2191 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786512017613366-32144-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786512017613366-32144-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786512017613366-32144-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 05:20:23.828136 2191 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 05:20:23.829681 2191 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 05:20:23.829805 2191 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 05:20:23.830259 2191 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 05:20:23.830296 2191 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 05:20:23.830322 2191 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 05:20:23.830338 2191 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 05:20:23.836576 2191 rpc_server.cc:307] RPC server started. Bound to: 127.31.100.3:37953
I20260812 05:20:23.836637 2404 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.100.3:37953 every 8 connection(s)
Aug 12 05:20:23 dist-test-slave-42z9 krb5kdc[1515](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512023, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.100.3@KRBTEST.COM for kudu/127.31.100.62@KRBTEST.COM
I20260812 05:20:23.836951 2191 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786512017613366-32144-0/minicluster-data/ts-2/data/info.pb
I20260812 05:20:23.837508 32144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0waFvT/build/release/bin/kudu as pid 2191
I20260812 05:20:23.837604 32144 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786512017613366-32144-0/minicluster-data/ts-2/wal/instance
I20260812 05:20:23.837791 32144 external_mini_cluster.cc:1428] Setting key 9b0b2913171bfc382f296e3839d4cb59
I20260812 05:20:23.840062 1676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:23.830619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.100.3:41357 (local address 127.31.100.62:35317)
0812 05:20:23.830682 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0812 05:20:23.830684 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 05:20:23.830830 (+ 146us) server_negotiation.cc:407] Connection header received
0812 05:20:23.831228 (+ 398us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:20:23.831230 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:20:23.831242 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:20:23.831317 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:20:23.831987 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:23.832746 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:23.834169 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:23.834472 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:23.834541 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:23.837349 (+ 2808us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 05:20:23.837351 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 05:20:23.837351 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 05:20:23.837359 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0812 05:20:23.838783 (+ 1424us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:20:23.839198 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:20:23.839200 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:20:23.839200 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:20:23.839258 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:20:23.839483 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:20:23.839484 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:20:23.839484 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:20:23.839594 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 05:20:23.839648 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0812 05:20:23.839891 (+ 243us) server_negotiation.cc:299] Negotiation successful
0812 05:20:23.839976 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260812 05:20:23.840202 2224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:23.830581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.62:35317 (local address 127.31.100.3:41357)
0812 05:20:23.830716 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0812 05:20:23.830752 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0812 05:20:23.831145 (+ 393us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:20:23.831364 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:20:23.831367 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:20:23.831636 (+ 269us) client_negotiation.cc:189] Negotiated authn=SASL
0812 05:20:23.831888 (+ 252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:20:23.831894 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:20:23.832832 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:20:23.832833 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:20:23.834067 (+ 1234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:20:23.834071 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:20:23.834563 (+ 492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:20:23.834564 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:20:23.834723 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:23.835641 (+ 918us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 05:20:23.835648 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0812 05:20:23.837271 (+ 1623us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 05:20:23.838849 (+ 1578us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 05:20:23.838856 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 05:20:23.838858 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 05:20:23.839133 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 05:20:23.839309 (+ 176us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 05:20:23.839310 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 05:20:23.839310 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 05:20:23.839424 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 05:20:23.839648 (+ 224us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 05:20:23.839654 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 05:20:23.839777 (+ 123us) client_negotiation.cc:769] Sending connection context
0812 05:20:23.839860 (+ 83us) client_negotiation.cc:240] Negotiation successful
0812 05:20:23.840010 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":70,"threads_started":1}
I20260812 05:20:23.840622 2407 heartbeater.cc:347] Connected to a master server at 127.31.100.62:35317
I20260812 05:20:23.840699 2407 heartbeater.cc:464] Registering TS with master...
I20260812 05:20:23.840844 2407 heartbeater.cc:511] Master 127.31.100.62:35317 requested a full tablet report, sending...
I20260812 05:20:23.841204 1585 ts_manager.cc:194] Registered new tserver with Master: 2e7c1aeb46b547e39cddeae55b4bd2a4 (127.31.100.3:37953)
I20260812 05:20:23.841689 1585 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.100.3@KRBTEST.COM'} at 127.31.100.3:41357
I20260812 05:20:23.845989 1676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:23.841551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.100.3:56255 (local address 127.31.100.62:35317)
0812 05:20:23.841588 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0812 05:20:23.841590 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 05:20:23.841689 (+ 99us) server_negotiation.cc:407] Connection header received
0812 05:20:23.841714 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:20:23.841715 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:20:23.841723 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:20:23.841779 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:20:23.842115 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:23.842651 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:23.843261 (+ 610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:23.843424 (+ 163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:23.843465 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:23.844177 (+ 712us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 05:20:23.844178 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 05:20:23.844178 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 05:20:23.844182 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0812 05:20:23.845205 (+ 1023us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:20:23.845503 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:20:23.845504 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:20:23.845504 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:20:23.845553 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:20:23.845675 (+ 122us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:20:23.845675 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:20:23.845675 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:20:23.845754 (+ 79us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 05:20:23.845794 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0812 05:20:23.845875 (+ 81us) server_negotiation.cc:299] Negotiation successful
0812 05:20:23.845925 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260812 05:20:23.845990 2420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:23.841512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.62:35317 (local address 127.31.100.3:56255)
0812 05:20:23.841630 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0812 05:20:23.841638 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0812 05:20:23.841684 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:20:23.841818 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:20:23.841819 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:20:23.842000 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0812 05:20:23.842047 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:20:23.842049 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:20:23.842725 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:20:23.842726 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:20:23.843207 (+ 481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:20:23.843208 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:20:23.843468 (+ 260us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:20:23.843469 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:20:23.843516 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:23.843855 (+ 339us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 05:20:23.843859 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0812 05:20:23.844120 (+ 261us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 05:20:23.845264 (+ 1144us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 05:20:23.845265 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 05:20:23.845265 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 05:20:23.845422 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 05:20:23.845594 (+ 172us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 05:20:23.845594 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 05:20:23.845594 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 05:20:23.845643 (+ 49us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 05:20:23.845787 (+ 144us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 05:20:23.845788 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 05:20:23.845819 (+ 31us) client_negotiation.cc:769] Sending connection context
0812 05:20:23.845860 (+ 41us) client_negotiation.cc:240] Negotiation successful
0812 05:20:23.845932 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":63,"threads_started":1}
I20260812 05:20:23.849232 32144 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 12 05:20:23 dist-test-slave-42z9 krb5kdc[1515](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512023, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 12 05:20:23 dist-test-slave-42z9 krb5kdc[1515](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512023, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.31.100.62@KRBTEST.COM
I20260812 05:20:23.870757 1676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:23.864882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34604 (local address 127.31.100.62:35317)
0812 05:20:23.864927 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0812 05:20:23.864929 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 05:20:23.864936 (+ 7us) server_negotiation.cc:407] Connection header received
0812 05:20:23.864984 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:20:23.864986 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:20:23.864995 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:20:23.865039 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:20:23.865655 (+ 616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:23.866097 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:23.866763 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:23.866937 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:23.866988 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:23.868832 (+ 1844us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 05:20:23.868834 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 05:20:23.868834 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 05:20:23.868841 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 05:20:23.869835 (+ 994us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:20:23.870131 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:20:23.870132 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:20:23.870132 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:20:23.870178 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:20:23.870332 (+ 154us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:20:23.870333 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:20:23.870333 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:20:23.870425 (+ 92us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 05:20:23.870517 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0812 05:20:23.870607 (+ 90us) server_negotiation.cc:299] Negotiation successful
0812 05:20:23.870674 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260812 05:20:23.871950 1585 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34604:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 05:20:23.873507 1585 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 05:20:23.879452 2430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:23.875350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.2:35527 (local address 127.0.0.1:39704)
0812 05:20:23.875447 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0812 05:20:23.875485 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0812 05:20:23.875540 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:20:23.876007 (+ 467us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:20:23.876009 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:20:23.876012 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 05:20:23.876087 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:20:23.876089 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:20:23.877038 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:20:23.877039 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:20:23.878186 (+ 1147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:20:23.878189 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:20:23.879121 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:20:23.879122 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:20:23.879215 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:23.879230 (+ 15us) client_negotiation.cc:769] Sending connection context
0812 05:20:23.879298 (+ 68us) client_negotiation.cc:240] Negotiation successful
0812 05:20:23.879333 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":75,"mutex_wait_us":6,"thread_start_us":43,"threads_started":1}
I20260812 05:20:23.879470 2431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:23.875454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39704 (local address 127.31.100.2:35527)
0812 05:20:23.875804 (+ 350us) server_negotiation.cc:206] Beginning negotiation
0812 05:20:23.875808 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 05:20:23.875818 (+ 10us) server_negotiation.cc:407] Connection header received
0812 05:20:23.875856 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:20:23.875858 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:20:23.875874 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:20:23.875996 (+ 122us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 05:20:23.876258 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:23.876959 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:23.878275 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:23.879038 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:23.879108 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:23.879160 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0812 05:20:23.879322 (+ 162us) server_negotiation.cc:299] Negotiation successful
0812 05:20:23.879372 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":312,"thread_start_us":71,"threads_started":1}
I20260812 05:20:23.879609 2437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:23.875729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41672 (local address 127.31.100.3:37953)
0812 05:20:23.875886 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0812 05:20:23.875889 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 05:20:23.875941 (+ 52us) server_negotiation.cc:407] Connection header received
0812 05:20:23.876102 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:20:23.876104 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:20:23.876117 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:20:23.876187 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 05:20:23.876304 (+ 117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:23.877344 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:23.878369 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:23.879040 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:23.879106 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:23.879160 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0812 05:20:23.879491 (+ 331us) server_negotiation.cc:299] Negotiation successful
0812 05:20:23.879526 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":91,"threads_started":1}
I20260812 05:20:23.879650 2434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:23.875659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38296 (local address 127.31.100.1:42473)
0812 05:20:23.875827 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0812 05:20:23.875832 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 05:20:23.875842 (+ 10us) server_negotiation.cc:407] Connection header received
0812 05:20:23.875881 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:20:23.875883 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:20:23.875898 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:20:23.876161 (+ 263us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 05:20:23.876223 (+ 62us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:23.877345 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:23.878586 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:23.879072 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:23.879409 (+ 337us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:23.879475 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0812 05:20:23.879503 (+ 28us) server_negotiation.cc:299] Negotiation successful
0812 05:20:23.879546 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":88,"threads_started":1}
I20260812 05:20:23.879681 2432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:23.875537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.1:42473 (local address 127.0.0.1:38296)
0812 05:20:23.875625 (+ 88us) negotiation.cc:107] Waiting for socket to connect
0812 05:20:23.875635 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0812 05:20:23.875688 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:20:23.876010 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:20:23.876011 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:20:23.876013 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 05:20:23.876085 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:20:23.876087 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:20:23.877428 (+ 1341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:20:23.877429 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:20:23.878517 (+ 1088us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:20:23.878519 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:20:23.879161 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:20:23.879162 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:20:23.879214 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:23.879230 (+ 16us) client_negotiation.cc:769] Sending connection context
0812 05:20:23.879274 (+ 44us) client_negotiation.cc:240] Negotiation successful
0812 05:20:23.879322 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":72,"mutex_wait_us":29,"thread_start_us":40,"threads_started":1}
I20260812 05:20:23.879453 2433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:23.875653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.3:37953 (local address 127.0.0.1:41672)
0812 05:20:23.875885 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0812 05:20:23.875895 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0812 05:20:23.875964 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:20:23.876187 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:20:23.876188 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:20:23.876189 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 05:20:23.876229 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:20:23.876230 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:20:23.877462 (+ 1232us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:20:23.877463 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:20:23.878290 (+ 827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:20:23.878292 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:20:23.879196 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:20:23.879197 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:20:23.879250 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:23.879255 (+ 5us) client_negotiation.cc:769] Sending connection context
0812 05:20:23.879295 (+ 40us) client_negotiation.cc:240] Negotiation successful
0812 05:20:23.879338 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":221,"mutex_wait_us":18,"thread_start_us":40,"threads_started":1}
I20260812 05:20:23.880182 2073 tablet_service.cc:1467] Processing CreateTablet for tablet 03b86499151a4e2fbdfd6bbb9888930a (DEFAULT_TABLE table=test-table [id=50f5951a2d564c92948dd20ea82b3e3a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 05:20:23.880563 2073 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 03b86499151a4e2fbdfd6bbb9888930a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 05:20:23.880578 2287 tablet_service.cc:1467] Processing CreateTablet for tablet 03b86499151a4e2fbdfd6bbb9888930a (DEFAULT_TABLE table=test-table [id=50f5951a2d564c92948dd20ea82b3e3a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 05:20:23.880578 1827 tablet_service.cc:1467] Processing CreateTablet for tablet 03b86499151a4e2fbdfd6bbb9888930a (DEFAULT_TABLE table=test-table [id=50f5951a2d564c92948dd20ea82b3e3a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 05:20:23.880910 2287 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 03b86499151a4e2fbdfd6bbb9888930a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 05:20:23.880910 1827 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 03b86499151a4e2fbdfd6bbb9888930a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 05:20:23.883273 2438 tablet_bootstrap.cc:492] T 03b86499151a4e2fbdfd6bbb9888930a P 280276f5683543e0aeeebad99ea84a84: Bootstrap starting.
I20260812 05:20:23.883862 2438 tablet_bootstrap.cc:654] T 03b86499151a4e2fbdfd6bbb9888930a P 280276f5683543e0aeeebad99ea84a84: Neither blocks nor log segments found. Creating new log.
I20260812 05:20:23.884016 2441 tablet_bootstrap.cc:492] T 03b86499151a4e2fbdfd6bbb9888930a P 2e7c1aeb46b547e39cddeae55b4bd2a4: Bootstrap starting.
I20260812 05:20:23.884328 2438 log.cc:824] T 03b86499151a4e2fbdfd6bbb9888930a P 280276f5683543e0aeeebad99ea84a84: Log is configured to *not* fsync() on all Append() calls
I20260812 05:20:23.884403 2442 tablet_bootstrap.cc:492] T 03b86499151a4e2fbdfd6bbb9888930a P 770f563b32324129ae355bd292cfce2e: Bootstrap starting.
I20260812 05:20:23.884832 2441 tablet_bootstrap.cc:654] T 03b86499151a4e2fbdfd6bbb9888930a P 2e7c1aeb46b547e39cddeae55b4bd2a4: Neither blocks nor log segments found. Creating new log.
I20260812 05:20:23.884994 2442 tablet_bootstrap.cc:654] T 03b86499151a4e2fbdfd6bbb9888930a P 770f563b32324129ae355bd292cfce2e: Neither blocks nor log segments found. Creating new log.
I20260812 05:20:23.885141 2441 log.cc:824] T 03b86499151a4e2fbdfd6bbb9888930a P 2e7c1aeb46b547e39cddeae55b4bd2a4: Log is configured to *not* fsync() on all Append() calls
I20260812 05:20:23.885275 2442 log.cc:824] T 03b86499151a4e2fbdfd6bbb9888930a P 770f563b32324129ae355bd292cfce2e: Log is configured to *not* fsync() on all Append() calls
I20260812 05:20:23.885354 2438 tablet_bootstrap.cc:492] T 03b86499151a4e2fbdfd6bbb9888930a P 280276f5683543e0aeeebad99ea84a84: No bootstrap required, opened a new log
I20260812 05:20:23.885424 2438 ts_tablet_manager.cc:1397] T 03b86499151a4e2fbdfd6bbb9888930a P 280276f5683543e0aeeebad99ea84a84: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260812 05:20:23.885919 2441 tablet_bootstrap.cc:492] T 03b86499151a4e2fbdfd6bbb9888930a P 2e7c1aeb46b547e39cddeae55b4bd2a4: No bootstrap required, opened a new log
I20260812 05:20:23.885999 2441 ts_tablet_manager.cc:1397] T 03b86499151a4e2fbdfd6bbb9888930a P 2e7c1aeb46b547e39cddeae55b4bd2a4: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260812 05:20:23.886047 2442 tablet_bootstrap.cc:492] T 03b86499151a4e2fbdfd6bbb9888930a P 770f563b32324129ae355bd292cfce2e: No bootstrap required, opened a new log
I20260812 05:20:23.886098 2442 ts_tablet_manager.cc:1397] T 03b86499151a4e2fbdfd6bbb9888930a P 770f563b32324129ae355bd292cfce2e: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260812 05:20:23.887094 2438 raft_consensus.cc:348] T 03b86499151a4e2fbdfd6bbb9888930a P 280276f5683543e0aeeebad99ea84a84 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "280276f5683543e0aeeebad99ea84a84" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 35527 } } peers { permanent_uuid: "770f563b32324129ae355bd292cfce2e" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 42473 } } peers { permanent_uuid: "2e7c1aeb46b547e39cddeae55b4bd2a4" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 37953 } }
I20260812 05:20:23.887218 2438 raft_consensus.cc:374] T 03b86499151a4e2fbdfd6bbb9888930a P 280276f5683543e0aeeebad99ea84a84 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 05:20:23.887249 2438 raft_consensus.cc:729] T 03b86499151a4e2fbdfd6bbb9888930a P 280276f5683543e0aeeebad99ea84a84 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 280276f5683543e0aeeebad99ea84a84, State: Initialized, Role: FOLLOWER
I20260812 05:20:23.887372 2438 consensus_queue.cc:260] T 03b86499151a4e2fbdfd6bbb9888930a P 280276f5683543e0aeeebad99ea84a84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "280276f5683543e0aeeebad99ea84a84" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 35527 } } peers { permanent_uuid: "770f563b32324129ae355bd292cfce2e" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 42473 } } peers { permanent_uuid: "2e7c1aeb46b547e39cddeae55b4bd2a4" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 37953 } }
I20260812 05:20:23.887625 2438 ts_tablet_manager.cc:1428] T 03b86499151a4e2fbdfd6bbb9888930a P 280276f5683543e0aeeebad99ea84a84: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260812 05:20:23.887744 2173 heartbeater.cc:503] Master 127.31.100.62:35317 was elected leader, sending a full tablet report...
I20260812 05:20:23.887877 2441 raft_consensus.cc:348] T 03b86499151a4e2fbdfd6bbb9888930a P 2e7c1aeb46b547e39cddeae55b4bd2a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "280276f5683543e0aeeebad99ea84a84" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 35527 } } peers { permanent_uuid: "770f563b32324129ae355bd292cfce2e" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 42473 } } peers { permanent_uuid: "2e7c1aeb46b547e39cddeae55b4bd2a4" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 37953 } }
I20260812 05:20:23.887905 2442 raft_consensus.cc:348] T 03b86499151a4e2fbdfd6bbb9888930a P 770f563b32324129ae355bd292cfce2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "280276f5683543e0aeeebad99ea84a84" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 35527 } } peers { permanent_uuid: "770f563b32324129ae355bd292cfce2e" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 42473 } } peers { permanent_uuid: "2e7c1aeb46b547e39cddeae55b4bd2a4" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 37953 } }
I20260812 05:20:23.888042 2441 raft_consensus.cc:374] T 03b86499151a4e2fbdfd6bbb9888930a P 2e7c1aeb46b547e39cddeae55b4bd2a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 05:20:23.888042 2442 raft_consensus.cc:374] T 03b86499151a4e2fbdfd6bbb9888930a P 770f563b32324129ae355bd292cfce2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 05:20:23.888082 2441 raft_consensus.cc:729] T 03b86499151a4e2fbdfd6bbb9888930a P 2e7c1aeb46b547e39cddeae55b4bd2a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e7c1aeb46b547e39cddeae55b4bd2a4, State: Initialized, Role: FOLLOWER
I20260812 05:20:23.888082 2442 raft_consensus.cc:729] T 03b86499151a4e2fbdfd6bbb9888930a P 770f563b32324129ae355bd292cfce2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 770f563b32324129ae355bd292cfce2e, State: Initialized, Role: FOLLOWER
I20260812 05:20:23.888182 2441 consensus_queue.cc:260] T 03b86499151a4e2fbdfd6bbb9888930a P 2e7c1aeb46b547e39cddeae55b4bd2a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "280276f5683543e0aeeebad99ea84a84" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 35527 } } peers { permanent_uuid: "770f563b32324129ae355bd292cfce2e" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 42473 } } peers { permanent_uuid: "2e7c1aeb46b547e39cddeae55b4bd2a4" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 37953 } }
I20260812 05:20:23.888182 2442 consensus_queue.cc:260] T 03b86499151a4e2fbdfd6bbb9888930a P 770f563b32324129ae355bd292cfce2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "280276f5683543e0aeeebad99ea84a84" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 35527 } } peers { permanent_uuid: "770f563b32324129ae355bd292cfce2e" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 42473 } } peers { permanent_uuid: "2e7c1aeb46b547e39cddeae55b4bd2a4" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 37953 } }
I20260812 05:20:23.888432 2441 ts_tablet_manager.cc:1428] T 03b86499151a4e2fbdfd6bbb9888930a P 2e7c1aeb46b547e39cddeae55b4bd2a4: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260812 05:20:23.888432 2442 ts_tablet_manager.cc:1428] T 03b86499151a4e2fbdfd6bbb9888930a P 770f563b32324129ae355bd292cfce2e: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260812 05:20:23.888461 2407 heartbeater.cc:503] Master 127.31.100.62:35317 was elected leader, sending a full tablet report...
I20260812 05:20:23.888711 1944 heartbeater.cc:503] Master 127.31.100.62:35317 was elected leader, sending a full tablet report...
I20260812 05:20:23.890697 2447 raft_consensus.cc:482] T 03b86499151a4e2fbdfd6bbb9888930a P 770f563b32324129ae355bd292cfce2e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 05:20:23.890774 2447 raft_consensus.cc:504] T 03b86499151a4e2fbdfd6bbb9888930a P 770f563b32324129ae355bd292cfce2e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "280276f5683543e0aeeebad99ea84a84" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 35527 } } peers { permanent_uuid: "770f563b32324129ae355bd292cfce2e" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 42473 } } peers { permanent_uuid: "2e7c1aeb46b547e39cddeae55b4bd2a4" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 37953 } }
I20260812 05:20:23.891031 2447 leader_election.cc:288] T 03b86499151a4e2fbdfd6bbb9888930a P 770f563b32324129ae355bd292cfce2e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 280276f5683543e0aeeebad99ea84a84 (127.31.100.2:35527), 2e7c1aeb46b547e39cddeae55b4bd2a4 (127.31.100.3:37953)
I20260812 05:20:23.894344 2437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:23.891292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.100.1:59343 (local address 127.31.100.3:37953)
0812 05:20:23.891325 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0812 05:20:23.891326 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0812 05:20:23.891373 (+ 47us) server_negotiation.cc:407] Connection header received
0812 05:20:23.891427 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:20:23.891428 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:20:23.891435 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:20:23.891479 (+ 44us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 05:20:23.891595 (+ 116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:23.892253 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:23.893654 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:23.894071 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:23.894121 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:23.894127 (+ 6us) server_negotiation.cc:1035] Waiting for connection context
0812 05:20:23.894258 (+ 131us) server_negotiation.cc:299] Negotiation successful
0812 05:20:23.894294 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260812 05:20:23.894368 2452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:23.891218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.3:37953 (local address 127.31.100.1:59343)
0812 05:20:23.891325 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0812 05:20:23.891334 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0812 05:20:23.891384 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:20:23.891480 (+ 96us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:20:23.891481 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:20:23.891482 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 05:20:23.891544 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:20:23.891546 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:20:23.892319 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:20:23.892319 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:20:23.893576 (+ 1257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:20:23.893579 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:20:23.894130 (+ 551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:20:23.894131 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:20:23.894192 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:23.894195 (+ 3us) client_negotiation.cc:769] Sending connection context
0812 05:20:23.894244 (+ 49us) client_negotiation.cc:240] Negotiation successful
0812 05:20:23.894279 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":91,"mutex_wait_us":58,"thread_start_us":69,"threads_started":1}
I20260812 05:20:23.894712 2323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03b86499151a4e2fbdfd6bbb9888930a" candidate_uuid: "770f563b32324129ae355bd292cfce2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2e7c1aeb46b547e39cddeae55b4bd2a4" is_pre_election: true
I20260812 05:20:23.894824 2323 raft_consensus.cc:2454] T 03b86499151a4e2fbdfd6bbb9888930a P 2e7c1aeb46b547e39cddeae55b4bd2a4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 770f563b32324129ae355bd292cfce2e in term 0.
I20260812 05:20:23.895040 1738 leader_election.cc:302] T 03b86499151a4e2fbdfd6bbb9888930a P 770f563b32324129ae355bd292cfce2e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2e7c1aeb46b547e39cddeae55b4bd2a4, 770f563b32324129ae355bd292cfce2e; no voters:
I20260812 05:20:23.895143 2447 raft_consensus.cc:2781] T 03b86499151a4e2fbdfd6bbb9888930a P 770f563b32324129ae355bd292cfce2e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 05:20:23.895192 2447 raft_consensus.cc:482] T 03b86499151a4e2fbdfd6bbb9888930a P 770f563b32324129ae355bd292cfce2e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 05:20:23.895218 2447 raft_consensus.cc:3037] T 03b86499151a4e2fbdfd6bbb9888930a P 770f563b32324129ae355bd292cfce2e [term 0 FOLLOWER]: Advancing to term 1
I20260812 05:20:23.895326 1751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:23.891136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.2:35527 (local address 127.31.100.1:54177)
0812 05:20:23.891171 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0812 05:20:23.891181 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0812 05:20:23.891222 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:20:23.891422 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:20:23.891423 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:20:23.891424 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 05:20:23.891484 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:20:23.891485 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:20:23.892319 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:20:23.892319 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:20:23.893576 (+ 1257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:20:23.893579 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:20:23.894668 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:20:23.894669 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:20:23.895164 (+ 495us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:23.895168 (+ 4us) client_negotiation.cc:769] Sending connection context
0812 05:20:23.895222 (+ 54us) client_negotiation.cc:240] Negotiation successful
0812 05:20:23.895259 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17,"mutex_wait_us":49}
I20260812 05:20:23.895334 2431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:23.891288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.100.1:54177 (local address 127.31.100.2:35527)
0812 05:20:23.891324 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0812 05:20:23.891326 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 05:20:23.891333 (+ 7us) server_negotiation.cc:407] Connection header received
0812 05:20:23.891351 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:20:23.891352 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:20:23.891359 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:20:23.891409 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 05:20:23.891542 (+ 133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:23.892255 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:23.894155 (+ 1900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:23.894591 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:23.894671 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:23.894679 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0812 05:20:23.895241 (+ 562us) server_negotiation.cc:299] Negotiation successful
0812 05:20:23.895283 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260812 05:20:23.895637 2103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03b86499151a4e2fbdfd6bbb9888930a" candidate_uuid: "770f563b32324129ae355bd292cfce2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "280276f5683543e0aeeebad99ea84a84" is_pre_election: true
I20260812 05:20:23.895728 2103 raft_consensus.cc:2454] T 03b86499151a4e2fbdfd6bbb9888930a P 280276f5683543e0aeeebad99ea84a84 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 770f563b32324129ae355bd292cfce2e in term 0.
I20260812 05:20:23.895890 2447 raft_consensus.cc:504] T 03b86499151a4e2fbdfd6bbb9888930a P 770f563b32324129ae355bd292cfce2e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "280276f5683543e0aeeebad99ea84a84" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 35527 } } peers { permanent_uuid: "770f563b32324129ae355bd292cfce2e" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 42473 } } peers { permanent_uuid: "2e7c1aeb46b547e39cddeae55b4bd2a4" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 37953 } }
I20260812 05:20:23.896003 2447 leader_election.cc:288] T 03b86499151a4e2fbdfd6bbb9888930a P 770f563b32324129ae355bd292cfce2e [CANDIDATE]: Term 1 election: Requested vote from peers 280276f5683543e0aeeebad99ea84a84 (127.31.100.2:35527), 2e7c1aeb46b547e39cddeae55b4bd2a4 (127.31.100.3:37953)
I20260812 05:20:23.896150 2103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03b86499151a4e2fbdfd6bbb9888930a" candidate_uuid: "770f563b32324129ae355bd292cfce2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "280276f5683543e0aeeebad99ea84a84"
I20260812 05:20:23.896157 2323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03b86499151a4e2fbdfd6bbb9888930a" candidate_uuid: "770f563b32324129ae355bd292cfce2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2e7c1aeb46b547e39cddeae55b4bd2a4"
I20260812 05:20:23.896211 2103 raft_consensus.cc:3037] T 03b86499151a4e2fbdfd6bbb9888930a P 280276f5683543e0aeeebad99ea84a84 [term 0 FOLLOWER]: Advancing to term 1
I20260812 05:20:23.896227 2323 raft_consensus.cc:3037] T 03b86499151a4e2fbdfd6bbb9888930a P 2e7c1aeb46b547e39cddeae55b4bd2a4 [term 0 FOLLOWER]: Advancing to term 1
I20260812 05:20:23.896780 2103 raft_consensus.cc:2454] T 03b86499151a4e2fbdfd6bbb9888930a P 280276f5683543e0aeeebad99ea84a84 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 770f563b32324129ae355bd292cfce2e in term 1.
I20260812 05:20:23.896932 1738 leader_election.cc:302] T 03b86499151a4e2fbdfd6bbb9888930a P 770f563b32324129ae355bd292cfce2e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 280276f5683543e0aeeebad99ea84a84, 770f563b32324129ae355bd292cfce2e; no voters:
I20260812 05:20:23.897014 2447 raft_consensus.cc:2781] T 03b86499151a4e2fbdfd6bbb9888930a P 770f563b32324129ae355bd292cfce2e [term 1 FOLLOWER]: Leader election won for term 1
I20260812 05:20:23.897080 2323 raft_consensus.cc:2454] T 03b86499151a4e2fbdfd6bbb9888930a P 2e7c1aeb46b547e39cddeae55b4bd2a4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 770f563b32324129ae355bd292cfce2e in term 1.
I20260812 05:20:23.897162 2447 raft_consensus.cc:686] T 03b86499151a4e2fbdfd6bbb9888930a P 770f563b32324129ae355bd292cfce2e [term 1 LEADER]: Becoming Leader. State: Replica: 770f563b32324129ae355bd292cfce2e, State: Running, Role: LEADER
I20260812 05:20:23.897240 2447 consensus_queue.cc:237] T 03b86499151a4e2fbdfd6bbb9888930a P 770f563b32324129ae355bd292cfce2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "280276f5683543e0aeeebad99ea84a84" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 35527 } } peers { permanent_uuid: "770f563b32324129ae355bd292cfce2e" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 42473 } } peers { permanent_uuid: "2e7c1aeb46b547e39cddeae55b4bd2a4" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 37953 } }
I20260812 05:20:23.897917 1585 catalog_manager.cc:5495] T 03b86499151a4e2fbdfd6bbb9888930a P 770f563b32324129ae355bd292cfce2e reported cstate change: term changed from 0 to 1, leader changed from <none> to 770f563b32324129ae355bd292cfce2e (127.31.100.1). New cstate: current_term: 1 leader_uuid: "770f563b32324129ae355bd292cfce2e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "280276f5683543e0aeeebad99ea84a84" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 35527 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "770f563b32324129ae355bd292cfce2e" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 42473 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2e7c1aeb46b547e39cddeae55b4bd2a4" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 37953 } health_report { overall_health: UNKNOWN } } }
I20260812 05:20:23.902998 2434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:23.900954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38312 (local address 127.31.100.1:42473)
0812 05:20:23.900989 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0812 05:20:23.900990 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0812 05:20:23.901001 (+ 11us) server_negotiation.cc:407] Connection header received
0812 05:20:23.901071 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:20:23.901072 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:20:23.901085 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:20:23.901135 (+ 50us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 05:20:23.901224 (+ 89us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:23.901718 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:23.902422 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:23.902593 (+ 171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:23.902637 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:23.902770 (+ 133us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 05:20:23.902846 (+ 76us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 05:20:23.902897 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0812 05:20:23.902925 (+ 28us) server_negotiation.cc:299] Negotiation successful
0812 05:20:23.902940 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260812 05:20:23.904382 2103 raft_consensus.cc:1261] T 03b86499151a4e2fbdfd6bbb9888930a P 280276f5683543e0aeeebad99ea84a84 [term 1 FOLLOWER]: Refusing update from remote peer 770f563b32324129ae355bd292cfce2e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 05:20:23.904384 2323 raft_consensus.cc:1261] T 03b86499151a4e2fbdfd6bbb9888930a P 2e7c1aeb46b547e39cddeae55b4bd2a4 [term 1 FOLLOWER]: Refusing update from remote peer 770f563b32324129ae355bd292cfce2e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 05:20:23.904628 2447 consensus_queue.cc:1035] T 03b86499151a4e2fbdfd6bbb9888930a P 770f563b32324129ae355bd292cfce2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "280276f5683543e0aeeebad99ea84a84" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 35527 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 05:20:23.904740 2453 consensus_queue.cc:1035] T 03b86499151a4e2fbdfd6bbb9888930a P 770f563b32324129ae355bd292cfce2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "2e7c1aeb46b547e39cddeae55b4bd2a4" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 37953 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 05:20:23.907826 2457 mvcc.cc:205] Tried to move back new op lower bound from 7317553249910288384 to 7317553249883652096. Current Snapshot: MvccSnapshot[applied={T|T < 7317553249910288384}]
I20260812 05:20:23.910032 1585 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34604:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 05:20:23.910140 1585 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34604:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 05:20:23.910840 1585 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3d208a9824f844afb0da31ac8bb9bc3a: Sending DeleteTablet for 3 replicas of tablet 03b86499151a4e2fbdfd6bbb9888930a
I20260812 05:20:23.911262 1827 tablet_service.cc:1514] Processing DeleteTablet for tablet 03b86499151a4e2fbdfd6bbb9888930a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 05:20:23 UTC) from {username='kudu', principal='kudu/127.31.100.62@KRBTEST.COM'} at 127.0.0.1:38296
I20260812 05:20:23.911576 2471 tablet_replica.cc:330] T 03b86499151a4e2fbdfd6bbb9888930a P 770f563b32324129ae355bd292cfce2e: stopping tablet replica
I20260812 05:20:23.911651 2471 raft_consensus.cc:2229] T 03b86499151a4e2fbdfd6bbb9888930a P 770f563b32324129ae355bd292cfce2e [term 1 LEADER]: Raft consensus shutting down.
I20260812 05:20:23.911732 2471 raft_consensus.cc:2258] T 03b86499151a4e2fbdfd6bbb9888930a P 770f563b32324129ae355bd292cfce2e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 05:20:23.913415 2073 tablet_service.cc:1514] Processing DeleteTablet for tablet 03b86499151a4e2fbdfd6bbb9888930a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 05:20:23 UTC) from {username='kudu', principal='kudu/127.31.100.62@KRBTEST.COM'} at 127.0.0.1:39704
I20260812 05:20:23.913676 2287 tablet_service.cc:1514] Processing DeleteTablet for tablet 03b86499151a4e2fbdfd6bbb9888930a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 05:20:23 UTC) from {username='kudu', principal='kudu/127.31.100.62@KRBTEST.COM'} at 127.0.0.1:41672
I20260812 05:20:23.913740 1585 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34604:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260812 05:20:23.913930 2472 tablet_replica.cc:330] T 03b86499151a4e2fbdfd6bbb9888930a P 280276f5683543e0aeeebad99ea84a84: stopping tablet replica
W20260812 05:20:23.913990 1585 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 05:20:23.914041 2471 ts_tablet_manager.cc:1905] T 03b86499151a4e2fbdfd6bbb9888930a P 770f563b32324129ae355bd292cfce2e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 05:20:23.914042 2472 raft_consensus.cc:2229] T 03b86499151a4e2fbdfd6bbb9888930a P 280276f5683543e0aeeebad99ea84a84 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 05:20:23.914141 2472 raft_consensus.cc:2258] T 03b86499151a4e2fbdfd6bbb9888930a P 280276f5683543e0aeeebad99ea84a84 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 05:20:23.914558 2472 ts_tablet_manager.cc:1905] T 03b86499151a4e2fbdfd6bbb9888930a P 280276f5683543e0aeeebad99ea84a84: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 05:20:23.915297 2073 tablet_service.cc:1467] Processing CreateTablet for tablet 2ae73929604d484583fac8ed20620889 (DEFAULT_TABLE table=test-table [id=1007f7c68ded45bba9685132b892472c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 05:20:23.915369 1827 tablet_service.cc:1467] Processing CreateTablet for tablet 2ae73929604d484583fac8ed20620889 (DEFAULT_TABLE table=test-table [id=1007f7c68ded45bba9685132b892472c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 05:20:23.915418 2073 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ae73929604d484583fac8ed20620889. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 05:20:23.915454 1827 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ae73929604d484583fac8ed20620889. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 05:20:23.916610 2442 tablet_bootstrap.cc:492] T 2ae73929604d484583fac8ed20620889 P 770f563b32324129ae355bd292cfce2e: Bootstrap starting.
I20260812 05:20:23.916666 2438 tablet_bootstrap.cc:492] T 2ae73929604d484583fac8ed20620889 P 280276f5683543e0aeeebad99ea84a84: Bootstrap starting.
I20260812 05:20:23.916659 2287 tablet_service.cc:1467] Processing CreateTablet for tablet 2ae73929604d484583fac8ed20620889 (DEFAULT_TABLE table=test-table [id=1007f7c68ded45bba9685132b892472c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 05:20:23.916756 2287 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ae73929604d484583fac8ed20620889. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 05:20:23.917065 2438 tablet_bootstrap.cc:654] T 2ae73929604d484583fac8ed20620889 P 280276f5683543e0aeeebad99ea84a84: Neither blocks nor log segments found. Creating new log.
I20260812 05:20:23.917161 2442 tablet_bootstrap.cc:654] T 2ae73929604d484583fac8ed20620889 P 770f563b32324129ae355bd292cfce2e: Neither blocks nor log segments found. Creating new log.
I20260812 05:20:23.918008 2441 tablet_bootstrap.cc:492] T 2ae73929604d484583fac8ed20620889 P 2e7c1aeb46b547e39cddeae55b4bd2a4: Bootstrap starting.
I20260812 05:20:23.918571 2441 tablet_bootstrap.cc:654] T 2ae73929604d484583fac8ed20620889 P 2e7c1aeb46b547e39cddeae55b4bd2a4: Neither blocks nor log segments found. Creating new log.
I20260812 05:20:23.918609 2472 ts_tablet_manager.cc:1918] T 03b86499151a4e2fbdfd6bbb9888930a P 280276f5683543e0aeeebad99ea84a84: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 05:20:23.918672 2472 log.cc:1196] T 03b86499151a4e2fbdfd6bbb9888930a P 280276f5683543e0aeeebad99ea84a84: Deleting WAL directory at /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786512017613366-32144-0/minicluster-data/ts-1/wal/wals/03b86499151a4e2fbdfd6bbb9888930a
I20260812 05:20:23.918901 2472 ts_tablet_manager.cc:1939] T 03b86499151a4e2fbdfd6bbb9888930a P 280276f5683543e0aeeebad99ea84a84: Deleting consensus metadata
I20260812 05:20:23.919283 1564 catalog_manager.cc:4841] TS 280276f5683543e0aeeebad99ea84a84 (127.31.100.2:35527): tablet 03b86499151a4e2fbdfd6bbb9888930a (table test-table [id=50f5951a2d564c92948dd20ea82b3e3a]) successfully deleted
I20260812 05:20:23.919333 2471 ts_tablet_manager.cc:1918] T 03b86499151a4e2fbdfd6bbb9888930a P 770f563b32324129ae355bd292cfce2e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 05:20:23.919404 2471 log.cc:1196] T 03b86499151a4e2fbdfd6bbb9888930a P 770f563b32324129ae355bd292cfce2e: Deleting WAL directory at /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786512017613366-32144-0/minicluster-data/ts-0/wal/wals/03b86499151a4e2fbdfd6bbb9888930a
I20260812 05:20:23.919648 2471 ts_tablet_manager.cc:1939] T 03b86499151a4e2fbdfd6bbb9888930a P 770f563b32324129ae355bd292cfce2e: Deleting consensus metadata
I20260812 05:20:23.919759 2438 tablet_bootstrap.cc:492] T 2ae73929604d484583fac8ed20620889 P 280276f5683543e0aeeebad99ea84a84: No bootstrap required, opened a new log
I20260812 05:20:23.919806 2438 ts_tablet_manager.cc:1397] T 2ae73929604d484583fac8ed20620889 P 280276f5683543e0aeeebad99ea84a84: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260812 05:20:23.919945 2438 raft_consensus.cc:348] T 2ae73929604d484583fac8ed20620889 P 280276f5683543e0aeeebad99ea84a84 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "770f563b32324129ae355bd292cfce2e" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 42473 } } peers { permanent_uuid: "280276f5683543e0aeeebad99ea84a84" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 35527 } } peers { permanent_uuid: "2e7c1aeb46b547e39cddeae55b4bd2a4" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 37953 } }
I20260812 05:20:23.920017 2438 raft_consensus.cc:374] T 2ae73929604d484583fac8ed20620889 P 280276f5683543e0aeeebad99ea84a84 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 05:20:23.920035 2438 raft_consensus.cc:729] T 2ae73929604d484583fac8ed20620889 P 280276f5683543e0aeeebad99ea84a84 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 280276f5683543e0aeeebad99ea84a84, State: Initialized, Role: FOLLOWER
I20260812 05:20:23.920032 1564 catalog_manager.cc:4841] TS 770f563b32324129ae355bd292cfce2e (127.31.100.1:42473): tablet 03b86499151a4e2fbdfd6bbb9888930a (table test-table [id=50f5951a2d564c92948dd20ea82b3e3a]) successfully deleted
I20260812 05:20:23.920071 2438 consensus_queue.cc:260] T 2ae73929604d484583fac8ed20620889 P 280276f5683543e0aeeebad99ea84a84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "770f563b32324129ae355bd292cfce2e" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 42473 } } peers { permanent_uuid: "280276f5683543e0aeeebad99ea84a84" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 35527 } } peers { permanent_uuid: "2e7c1aeb46b547e39cddeae55b4bd2a4" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 37953 } }
I20260812 05:20:23.920149 2438 ts_tablet_manager.cc:1428] T 2ae73929604d484583fac8ed20620889 P 280276f5683543e0aeeebad99ea84a84: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260812 05:20:23.920241 2441 tablet_bootstrap.cc:492] T 2ae73929604d484583fac8ed20620889 P 2e7c1aeb46b547e39cddeae55b4bd2a4: No bootstrap required, opened a new log
I20260812 05:20:23.920287 2441 ts_tablet_manager.cc:1397] T 2ae73929604d484583fac8ed20620889 P 2e7c1aeb46b547e39cddeae55b4bd2a4: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260812 05:20:23.920429 2441 raft_consensus.cc:348] T 2ae73929604d484583fac8ed20620889 P 2e7c1aeb46b547e39cddeae55b4bd2a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "770f563b32324129ae355bd292cfce2e" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 42473 } } peers { permanent_uuid: "280276f5683543e0aeeebad99ea84a84" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 35527 } } peers { permanent_uuid: "2e7c1aeb46b547e39cddeae55b4bd2a4" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 37953 } }
I20260812 05:20:23.920508 2441 raft_consensus.cc:374] T 2ae73929604d484583fac8ed20620889 P 2e7c1aeb46b547e39cddeae55b4bd2a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 05:20:23.920526 2441 raft_consensus.cc:729] T 2ae73929604d484583fac8ed20620889 P 2e7c1aeb46b547e39cddeae55b4bd2a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e7c1aeb46b547e39cddeae55b4bd2a4, State: Initialized, Role: FOLLOWER
I20260812 05:20:23.920565 2441 consensus_queue.cc:260] T 2ae73929604d484583fac8ed20620889 P 2e7c1aeb46b547e39cddeae55b4bd2a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "770f563b32324129ae355bd292cfce2e" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 42473 } } peers { permanent_uuid: "280276f5683543e0aeeebad99ea84a84" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 35527 } } peers { permanent_uuid: "2e7c1aeb46b547e39cddeae55b4bd2a4" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 37953 } }
I20260812 05:20:23.920648 2441 ts_tablet_manager.cc:1428] T 2ae73929604d484583fac8ed20620889 P 2e7c1aeb46b547e39cddeae55b4bd2a4: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260812 05:20:23.920753 2474 tablet_replica.cc:330] T 03b86499151a4e2fbdfd6bbb9888930a P 2e7c1aeb46b547e39cddeae55b4bd2a4: stopping tablet replica
I20260812 05:20:23.920831 2474 raft_consensus.cc:2229] T 03b86499151a4e2fbdfd6bbb9888930a P 2e7c1aeb46b547e39cddeae55b4bd2a4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 05:20:23.920928 2474 raft_consensus.cc:2258] T 03b86499151a4e2fbdfd6bbb9888930a P 2e7c1aeb46b547e39cddeae55b4bd2a4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 05:20:23.921365 2442 tablet_bootstrap.cc:492] T 2ae73929604d484583fac8ed20620889 P 770f563b32324129ae355bd292cfce2e: No bootstrap required, opened a new log
I20260812 05:20:23.921411 2442 ts_tablet_manager.cc:1397] T 2ae73929604d484583fac8ed20620889 P 770f563b32324129ae355bd292cfce2e: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20260812 05:20:23.921590 2442 raft_consensus.cc:348] T 2ae73929604d484583fac8ed20620889 P 770f563b32324129ae355bd292cfce2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "770f563b32324129ae355bd292cfce2e" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 42473 } } peers { permanent_uuid: "280276f5683543e0aeeebad99ea84a84" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 35527 } } peers { permanent_uuid: "2e7c1aeb46b547e39cddeae55b4bd2a4" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 37953 } }
I20260812 05:20:23.921665 2442 raft_consensus.cc:374] T 2ae73929604d484583fac8ed20620889 P 770f563b32324129ae355bd292cfce2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 05:20:23.921685 2442 raft_consensus.cc:729] T 2ae73929604d484583fac8ed20620889 P 770f563b32324129ae355bd292cfce2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 770f563b32324129ae355bd292cfce2e, State: Initialized, Role: FOLLOWER
I20260812 05:20:23.921728 2442 consensus_queue.cc:260] T 2ae73929604d484583fac8ed20620889 P 770f563b32324129ae355bd292cfce2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "770f563b32324129ae355bd292cfce2e" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 42473 } } peers { permanent_uuid: "280276f5683543e0aeeebad99ea84a84" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 35527 } } peers { permanent_uuid: "2e7c1aeb46b547e39cddeae55b4bd2a4" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 37953 } }
I20260812 05:20:23.921803 2442 ts_tablet_manager.cc:1428] T 2ae73929604d484583fac8ed20620889 P 770f563b32324129ae355bd292cfce2e: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260812 05:20:23.922436 2474 ts_tablet_manager.cc:1905] T 03b86499151a4e2fbdfd6bbb9888930a P 2e7c1aeb46b547e39cddeae55b4bd2a4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 05:20:23.923542 2474 ts_tablet_manager.cc:1918] T 03b86499151a4e2fbdfd6bbb9888930a P 2e7c1aeb46b547e39cddeae55b4bd2a4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 05:20:23.923600 2474 log.cc:1196] T 03b86499151a4e2fbdfd6bbb9888930a P 2e7c1aeb46b547e39cddeae55b4bd2a4: Deleting WAL directory at /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786512017613366-32144-0/minicluster-data/ts-2/wal/wals/03b86499151a4e2fbdfd6bbb9888930a
I20260812 05:20:23.923787 2474 ts_tablet_manager.cc:1939] T 03b86499151a4e2fbdfd6bbb9888930a P 2e7c1aeb46b547e39cddeae55b4bd2a4: Deleting consensus metadata
I20260812 05:20:23.924124 1564 catalog_manager.cc:4841] TS 2e7c1aeb46b547e39cddeae55b4bd2a4 (127.31.100.3:37953): tablet 03b86499151a4e2fbdfd6bbb9888930a (table test-table [id=50f5951a2d564c92948dd20ea82b3e3a]) successfully deleted
W20260812 05:20:23.933660 2174 tablet.cc:2369] T 2ae73929604d484583fac8ed20620889 P 280276f5683543e0aeeebad99ea84a84: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 05:20:23.949476 2447 raft_consensus.cc:482] T 2ae73929604d484583fac8ed20620889 P 770f563b32324129ae355bd292cfce2e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 05:20:23.949546 2447 raft_consensus.cc:504] T 2ae73929604d484583fac8ed20620889 P 770f563b32324129ae355bd292cfce2e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "770f563b32324129ae355bd292cfce2e" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 42473 } } peers { permanent_uuid: "280276f5683543e0aeeebad99ea84a84" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 35527 } } peers { permanent_uuid: "2e7c1aeb46b547e39cddeae55b4bd2a4" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 37953 } }
I20260812 05:20:23.949666 2447 leader_election.cc:288] T 2ae73929604d484583fac8ed20620889 P 770f563b32324129ae355bd292cfce2e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 280276f5683543e0aeeebad99ea84a84 (127.31.100.2:35527), 2e7c1aeb46b547e39cddeae55b4bd2a4 (127.31.100.3:37953)
I20260812 05:20:23.949818 2103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ae73929604d484583fac8ed20620889" candidate_uuid: "770f563b32324129ae355bd292cfce2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "280276f5683543e0aeeebad99ea84a84" is_pre_election: true
I20260812 05:20:23.949903 2103 raft_consensus.cc:2454] T 2ae73929604d484583fac8ed20620889 P 280276f5683543e0aeeebad99ea84a84 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 770f563b32324129ae355bd292cfce2e in term 0.
I20260812 05:20:23.949924 2323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ae73929604d484583fac8ed20620889" candidate_uuid: "770f563b32324129ae355bd292cfce2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2e7c1aeb46b547e39cddeae55b4bd2a4" is_pre_election: true
I20260812 05:20:23.950019 2323 raft_consensus.cc:2454] T 2ae73929604d484583fac8ed20620889 P 2e7c1aeb46b547e39cddeae55b4bd2a4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 770f563b32324129ae355bd292cfce2e in term 0.
I20260812 05:20:23.950161 1738 leader_election.cc:302] T 2ae73929604d484583fac8ed20620889 P 770f563b32324129ae355bd292cfce2e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 280276f5683543e0aeeebad99ea84a84, 770f563b32324129ae355bd292cfce2e; no voters:
I20260812 05:20:23.950276 2447 raft_consensus.cc:2781] T 2ae73929604d484583fac8ed20620889 P 770f563b32324129ae355bd292cfce2e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 05:20:23.950330 2447 raft_consensus.cc:482] T 2ae73929604d484583fac8ed20620889 P 770f563b32324129ae355bd292cfce2e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 05:20:23.950357 2447 raft_consensus.cc:3037] T 2ae73929604d484583fac8ed20620889 P 770f563b32324129ae355bd292cfce2e [term 0 FOLLOWER]: Advancing to term 1
I20260812 05:20:23.951123 2447 raft_consensus.cc:504] T 2ae73929604d484583fac8ed20620889 P 770f563b32324129ae355bd292cfce2e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "770f563b32324129ae355bd292cfce2e" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 42473 } } peers { permanent_uuid: "280276f5683543e0aeeebad99ea84a84" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 35527 } } peers { permanent_uuid: "2e7c1aeb46b547e39cddeae55b4bd2a4" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 37953 } }
I20260812 05:20:23.951244 2447 leader_election.cc:288] T 2ae73929604d484583fac8ed20620889 P 770f563b32324129ae355bd292cfce2e [CANDIDATE]: Term 1 election: Requested vote from peers 280276f5683543e0aeeebad99ea84a84 (127.31.100.2:35527), 2e7c1aeb46b547e39cddeae55b4bd2a4 (127.31.100.3:37953)
I20260812 05:20:23.951440 2103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ae73929604d484583fac8ed20620889" candidate_uuid: "770f563b32324129ae355bd292cfce2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "280276f5683543e0aeeebad99ea84a84"
I20260812 05:20:23.951467 2323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ae73929604d484583fac8ed20620889" candidate_uuid: "770f563b32324129ae355bd292cfce2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2e7c1aeb46b547e39cddeae55b4bd2a4"
I20260812 05:20:23.951510 2103 raft_consensus.cc:3037] T 2ae73929604d484583fac8ed20620889 P 280276f5683543e0aeeebad99ea84a84 [term 0 FOLLOWER]: Advancing to term 1
I20260812 05:20:23.951536 2323 raft_consensus.cc:3037] T 2ae73929604d484583fac8ed20620889 P 2e7c1aeb46b547e39cddeae55b4bd2a4 [term 0 FOLLOWER]: Advancing to term 1
I20260812 05:20:23.952095 2103 raft_consensus.cc:2454] T 2ae73929604d484583fac8ed20620889 P 280276f5683543e0aeeebad99ea84a84 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 770f563b32324129ae355bd292cfce2e in term 1.
I20260812 05:20:23.952239 1738 leader_election.cc:302] T 2ae73929604d484583fac8ed20620889 P 770f563b32324129ae355bd292cfce2e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 280276f5683543e0aeeebad99ea84a84, 770f563b32324129ae355bd292cfce2e; no voters:
I20260812 05:20:23.952343 2323 raft_consensus.cc:2454] T 2ae73929604d484583fac8ed20620889 P 2e7c1aeb46b547e39cddeae55b4bd2a4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 770f563b32324129ae355bd292cfce2e in term 1.
I20260812 05:20:23.952550 2447 raft_consensus.cc:2781] T 2ae73929604d484583fac8ed20620889 P 770f563b32324129ae355bd292cfce2e [term 1 FOLLOWER]: Leader election won for term 1
I20260812 05:20:23.952606 2447 raft_consensus.cc:686] T 2ae73929604d484583fac8ed20620889 P 770f563b32324129ae355bd292cfce2e [term 1 LEADER]: Becoming Leader. State: Replica: 770f563b32324129ae355bd292cfce2e, State: Running, Role: LEADER
I20260812 05:20:23.952656 2447 consensus_queue.cc:237] T 2ae73929604d484583fac8ed20620889 P 770f563b32324129ae355bd292cfce2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "770f563b32324129ae355bd292cfce2e" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 42473 } } peers { permanent_uuid: "280276f5683543e0aeeebad99ea84a84" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 35527 } } peers { permanent_uuid: "2e7c1aeb46b547e39cddeae55b4bd2a4" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 37953 } }
I20260812 05:20:23.953115 1585 catalog_manager.cc:5495] T 2ae73929604d484583fac8ed20620889 P 770f563b32324129ae355bd292cfce2e reported cstate change: term changed from 0 to 1, leader changed from <none> to 770f563b32324129ae355bd292cfce2e (127.31.100.1). New cstate: current_term: 1 leader_uuid: "770f563b32324129ae355bd292cfce2e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "770f563b32324129ae355bd292cfce2e" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 42473 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "280276f5683543e0aeeebad99ea84a84" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 35527 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2e7c1aeb46b547e39cddeae55b4bd2a4" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 37953 } health_report { overall_health: UNKNOWN } } }
Aug 12 05:20:23 dist-test-slave-42z9 krb5kdc[1515](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512023, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.100.62@KRBTEST.COM for kudu/127.31.100.62@KRBTEST.COM
I20260812 05:20:23.997885 2488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:23.990094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.62:35317 (local address 127.0.0.1:34610)
0812 05:20:23.990411 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0812 05:20:23.990422 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0812 05:20:23.990503 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:20:23.990640 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:20:23.990641 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:20:23.991075 (+ 434us) client_negotiation.cc:189] Negotiated authn=SASL
0812 05:20:23.991135 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:20:23.991138 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:20:23.991804 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:20:23.991806 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:20:23.992639 (+ 833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:20:23.992642 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:20:23.993021 (+ 379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:20:23.993022 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:20:23.993087 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:23.993664 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 05:20:23.993670 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0812 05:20:23.995245 (+ 1575us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 05:20:23.996646 (+ 1401us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 05:20:23.996654 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 05:20:23.996656 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 05:20:23.996926 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 05:20:23.997109 (+ 183us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 05:20:23.997110 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 05:20:23.997110 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 05:20:23.997185 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 05:20:23.997534 (+ 349us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 05:20:23.997541 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 05:20:23.997589 (+ 48us) client_negotiation.cc:769] Sending connection context
0812 05:20:23.997653 (+ 64us) client_negotiation.cc:240] Negotiation successful
0812 05:20:23.997769 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":286,"thread_start_us":87,"threads_started":1}
I20260812 05:20:23.997927 1676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:23.990201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34610 (local address 127.31.100.62:35317)
0812 05:20:23.990273 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0812 05:20:23.990275 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 05:20:23.990520 (+ 245us) server_negotiation.cc:407] Connection header received
0812 05:20:23.990551 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:20:23.990552 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:20:23.990562 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:20:23.990617 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:20:23.991224 (+ 607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:23.991721 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:23.992732 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:23.992943 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:23.992998 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:23.995347 (+ 2349us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 05:20:23.995348 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 05:20:23.995348 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 05:20:23.995355 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 05:20:23.996563 (+ 1208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:20:23.996998 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:20:23.996999 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:20:23.996999 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:20:23.997048 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:20:23.997277 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:20:23.997278 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:20:23.997278 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:20:23.997392 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 05:20:23.997527 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0812 05:20:23.997680 (+ 153us) server_negotiation.cc:299] Negotiation successful
0812 05:20:23.997828 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":84,"server-negotiator.queue_time_us":23}
I20260812 05:20:23.999850 1585 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.31.100.62@KRBTEST.COM'} at 127.0.0.1:34610:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260812 05:20:24.000110 1585 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 05:20:24.001606 2073 tablet_service.cc:1467] Processing CreateTablet for tablet b748824c58bc4b36a1b64cd54618a5e4 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=90900e107a8b41baabb3834a309008c2]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260812 05:20:24.001557 1827 tablet_service.cc:1467] Processing CreateTablet for tablet b748824c58bc4b36a1b64cd54618a5e4 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=90900e107a8b41baabb3834a309008c2]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260812 05:20:24.001652 2287 tablet_service.cc:1467] Processing CreateTablet for tablet b748824c58bc4b36a1b64cd54618a5e4 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=90900e107a8b41baabb3834a309008c2]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260812 05:20:24.001770 2287 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b748824c58bc4b36a1b64cd54618a5e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 05:20:24.001771 2073 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b748824c58bc4b36a1b64cd54618a5e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 05:20:24.001772 1827 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b748824c58bc4b36a1b64cd54618a5e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 05:20:24.003237 2438 tablet_bootstrap.cc:492] T b748824c58bc4b36a1b64cd54618a5e4 P 280276f5683543e0aeeebad99ea84a84: Bootstrap starting.
I20260812 05:20:24.003264 2442 tablet_bootstrap.cc:492] T b748824c58bc4b36a1b64cd54618a5e4 P 770f563b32324129ae355bd292cfce2e: Bootstrap starting.
I20260812 05:20:24.003666 2438 tablet_bootstrap.cc:654] T b748824c58bc4b36a1b64cd54618a5e4 P 280276f5683543e0aeeebad99ea84a84: Neither blocks nor log segments found. Creating new log.
I20260812 05:20:24.003885 2442 tablet_bootstrap.cc:654] T b748824c58bc4b36a1b64cd54618a5e4 P 770f563b32324129ae355bd292cfce2e: Neither blocks nor log segments found. Creating new log.
I20260812 05:20:24.004016 2441 tablet_bootstrap.cc:492] T b748824c58bc4b36a1b64cd54618a5e4 P 2e7c1aeb46b547e39cddeae55b4bd2a4: Bootstrap starting.
I20260812 05:20:24.004540 2441 tablet_bootstrap.cc:654] T b748824c58bc4b36a1b64cd54618a5e4 P 2e7c1aeb46b547e39cddeae55b4bd2a4: Neither blocks nor log segments found. Creating new log.
I20260812 05:20:24.004652 2442 tablet_bootstrap.cc:492] T b748824c58bc4b36a1b64cd54618a5e4 P 770f563b32324129ae355bd292cfce2e: No bootstrap required, opened a new log
I20260812 05:20:24.004695 2442 ts_tablet_manager.cc:1397] T b748824c58bc4b36a1b64cd54618a5e4 P 770f563b32324129ae355bd292cfce2e: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260812 05:20:24.004835 2442 raft_consensus.cc:348] T b748824c58bc4b36a1b64cd54618a5e4 P 770f563b32324129ae355bd292cfce2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "770f563b32324129ae355bd292cfce2e" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 42473 } } peers { permanent_uuid: "280276f5683543e0aeeebad99ea84a84" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 35527 } } peers { permanent_uuid: "2e7c1aeb46b547e39cddeae55b4bd2a4" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 37953 } }
I20260812 05:20:24.004895 2442 raft_consensus.cc:374] T b748824c58bc4b36a1b64cd54618a5e4 P 770f563b32324129ae355bd292cfce2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 05:20:24.004925 2442 raft_consensus.cc:729] T b748824c58bc4b36a1b64cd54618a5e4 P 770f563b32324129ae355bd292cfce2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 770f563b32324129ae355bd292cfce2e, State: Initialized, Role: FOLLOWER
I20260812 05:20:24.004969 2442 consensus_queue.cc:260] T b748824c58bc4b36a1b64cd54618a5e4 P 770f563b32324129ae355bd292cfce2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "770f563b32324129ae355bd292cfce2e" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 42473 } } peers { permanent_uuid: "280276f5683543e0aeeebad99ea84a84" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 35527 } } peers { permanent_uuid: "2e7c1aeb46b547e39cddeae55b4bd2a4" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 37953 } }
I20260812 05:20:24.005059 2442 ts_tablet_manager.cc:1428] T b748824c58bc4b36a1b64cd54618a5e4 P 770f563b32324129ae355bd292cfce2e: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260812 05:20:24.005069 2438 tablet_bootstrap.cc:492] T b748824c58bc4b36a1b64cd54618a5e4 P 280276f5683543e0aeeebad99ea84a84: No bootstrap required, opened a new log
I20260812 05:20:24.005111 2438 ts_tablet_manager.cc:1397] T b748824c58bc4b36a1b64cd54618a5e4 P 280276f5683543e0aeeebad99ea84a84: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260812 05:20:24.005084 2453 tablet_replica.cc:439] T b748824c58bc4b36a1b64cd54618a5e4 P 770f563b32324129ae355bd292cfce2e: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "770f563b32324129ae355bd292cfce2e" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 42473 } } peers { permanent_uuid: "280276f5683543e0aeeebad99ea84a84" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 35527 } } peers { permanent_uuid: "2e7c1aeb46b547e39cddeae55b4bd2a4" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 37953 } } }
I20260812 05:20:24.005167 2453 tablet_replica.cc:442] T b748824c58bc4b36a1b64cd54618a5e4 P 770f563b32324129ae355bd292cfce2e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 05:20:24.005336 2438 raft_consensus.cc:348] T b748824c58bc4b36a1b64cd54618a5e4 P 280276f5683543e0aeeebad99ea84a84 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "770f563b32324129ae355bd292cfce2e" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 42473 } } peers { permanent_uuid: "280276f5683543e0aeeebad99ea84a84" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 35527 } } peers { permanent_uuid: "2e7c1aeb46b547e39cddeae55b4bd2a4" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 37953 } }
I20260812 05:20:24.005434 2438 raft_consensus.cc:374] T b748824c58bc4b36a1b64cd54618a5e4 P 280276f5683543e0aeeebad99ea84a84 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 05:20:24.005486 2438 raft_consensus.cc:729] T b748824c58bc4b36a1b64cd54618a5e4 P 280276f5683543e0aeeebad99ea84a84 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 280276f5683543e0aeeebad99ea84a84, State: Initialized, Role: FOLLOWER
I20260812 05:20:24.005532 2438 consensus_queue.cc:260] T b748824c58bc4b36a1b64cd54618a5e4 P 280276f5683543e0aeeebad99ea84a84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "770f563b32324129ae355bd292cfce2e" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 42473 } } peers { permanent_uuid: "280276f5683543e0aeeebad99ea84a84" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 35527 } } peers { permanent_uuid: "2e7c1aeb46b547e39cddeae55b4bd2a4" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 37953 } }
I20260812 05:20:24.005622 2438 ts_tablet_manager.cc:1428] T b748824c58bc4b36a1b64cd54618a5e4 P 280276f5683543e0aeeebad99ea84a84: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260812 05:20:24.005679 2446 tablet_replica.cc:439] T b748824c58bc4b36a1b64cd54618a5e4 P 280276f5683543e0aeeebad99ea84a84: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "770f563b32324129ae355bd292cfce2e" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 42473 } } peers { permanent_uuid: "280276f5683543e0aeeebad99ea84a84" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 35527 } } peers { permanent_uuid: "2e7c1aeb46b547e39cddeae55b4bd2a4" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 37953 } } }
I20260812 05:20:24.005762 2446 tablet_replica.cc:442] T b748824c58bc4b36a1b64cd54618a5e4 P 280276f5683543e0aeeebad99ea84a84: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 05:20:24.006183 2441 tablet_bootstrap.cc:492] T b748824c58bc4b36a1b64cd54618a5e4 P 2e7c1aeb46b547e39cddeae55b4bd2a4: No bootstrap required, opened a new log
I20260812 05:20:24.006227 2441 ts_tablet_manager.cc:1397] T b748824c58bc4b36a1b64cd54618a5e4 P 2e7c1aeb46b547e39cddeae55b4bd2a4: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260812 05:20:24.006553 2441 raft_consensus.cc:348] T b748824c58bc4b36a1b64cd54618a5e4 P 2e7c1aeb46b547e39cddeae55b4bd2a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "770f563b32324129ae355bd292cfce2e" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 42473 } } peers { permanent_uuid: "280276f5683543e0aeeebad99ea84a84" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 35527 } } peers { permanent_uuid: "2e7c1aeb46b547e39cddeae55b4bd2a4" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 37953 } }
I20260812 05:20:24.006613 2441 raft_consensus.cc:374] T b748824c58bc4b36a1b64cd54618a5e4 P 2e7c1aeb46b547e39cddeae55b4bd2a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 05:20:24.006624 2441 raft_consensus.cc:729] T b748824c58bc4b36a1b64cd54618a5e4 P 2e7c1aeb46b547e39cddeae55b4bd2a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e7c1aeb46b547e39cddeae55b4bd2a4, State: Initialized, Role: FOLLOWER
I20260812 05:20:24.006656 2441 consensus_queue.cc:260] T b748824c58bc4b36a1b64cd54618a5e4 P 2e7c1aeb46b547e39cddeae55b4bd2a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "770f563b32324129ae355bd292cfce2e" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 42473 } } peers { permanent_uuid: "280276f5683543e0aeeebad99ea84a84" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 35527 } } peers { permanent_uuid: "2e7c1aeb46b547e39cddeae55b4bd2a4" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 37953 } }
I20260812 05:20:24.006738 2441 ts_tablet_manager.cc:1428] T b748824c58bc4b36a1b64cd54618a5e4 P 2e7c1aeb46b547e39cddeae55b4bd2a4: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260812 05:20:24.006808 2451 tablet_replica.cc:439] T b748824c58bc4b36a1b64cd54618a5e4 P 2e7c1aeb46b547e39cddeae55b4bd2a4: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "770f563b32324129ae355bd292cfce2e" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 42473 } } peers { permanent_uuid: "280276f5683543e0aeeebad99ea84a84" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 35527 } } peers { permanent_uuid: "2e7c1aeb46b547e39cddeae55b4bd2a4" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 37953 } } }
I20260812 05:20:24.006883 2451 tablet_replica.cc:442] T b748824c58bc4b36a1b64cd54618a5e4 P 2e7c1aeb46b547e39cddeae55b4bd2a4: This TxnStatusTablet replica's current role is: FOLLOWER
W20260812 05:20:24.037149 1950 tablet.cc:2369] T 2ae73929604d484583fac8ed20620889 P 770f563b32324129ae355bd292cfce2e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 05:20:24.044811 2446 raft_consensus.cc:482] T b748824c58bc4b36a1b64cd54618a5e4 P 280276f5683543e0aeeebad99ea84a84 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 05:20:24.044895 2446 raft_consensus.cc:504] T b748824c58bc4b36a1b64cd54618a5e4 P 280276f5683543e0aeeebad99ea84a84 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "770f563b32324129ae355bd292cfce2e" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 42473 } } peers { permanent_uuid: "280276f5683543e0aeeebad99ea84a84" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 35527 } } peers { permanent_uuid: "2e7c1aeb46b547e39cddeae55b4bd2a4" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 37953 } }
I20260812 05:20:24.045190 2446 leader_election.cc:288] T b748824c58bc4b36a1b64cd54618a5e4 P 280276f5683543e0aeeebad99ea84a84 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 770f563b32324129ae355bd292cfce2e (127.31.100.1:42473), 2e7c1aeb46b547e39cddeae55b4bd2a4 (127.31.100.3:37953)
I20260812 05:20:24.048305 2434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:24.045410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.100.2:37409 (local address 127.31.100.1:42473)
0812 05:20:24.045478 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0812 05:20:24.045480 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 05:20:24.045491 (+ 11us) server_negotiation.cc:407] Connection header received
0812 05:20:24.045607 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:20:24.045608 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:20:24.045617 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:20:24.045673 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 05:20:24.045853 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:24.046378 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:24.047448 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:24.047975 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:24.048030 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:24.048038 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0812 05:20:24.048222 (+ 184us) server_negotiation.cc:299] Negotiation successful
0812 05:20:24.048256 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260812 05:20:24.048305 2003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:24.045351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.1:42473 (local address 127.31.100.2:37409)
0812 05:20:24.045405 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0812 05:20:24.045416 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0812 05:20:24.045486 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:20:24.045694 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:20:24.045696 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:20:24.045698 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 05:20:24.045773 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:20:24.045775 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:20:24.046456 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:20:24.046457 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:20:24.047331 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:20:24.047333 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:20:24.048050 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:20:24.048051 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:20:24.048131 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:24.048136 (+ 5us) client_negotiation.cc:769] Sending connection context
0812 05:20:24.048193 (+ 57us) client_negotiation.cc:240] Negotiation successful
0812 05:20:24.048232 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260812 05:20:24.048635 1861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b748824c58bc4b36a1b64cd54618a5e4" candidate_uuid: "280276f5683543e0aeeebad99ea84a84" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "770f563b32324129ae355bd292cfce2e" is_pre_election: true
I20260812 05:20:24.048756 2494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:24.045486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.3:37953 (local address 127.31.100.2:47417)
0812 05:20:24.045714 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0812 05:20:24.045722 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0812 05:20:24.045766 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:20:24.045879 (+ 113us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:20:24.045879 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:20:24.045881 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 05:20:24.045963 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:20:24.045965 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:20:24.046574 (+ 609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:20:24.046575 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:20:24.047872 (+ 1297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:20:24.047875 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:20:24.048546 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:20:24.048547 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:20:24.048623 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:24.048626 (+ 3us) client_negotiation.cc:769] Sending connection context
0812 05:20:24.048672 (+ 46us) client_negotiation.cc:240] Negotiation successful
0812 05:20:24.048700 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":158,"threads_started":1}
I20260812 05:20:24.048770 1861 raft_consensus.cc:2454] T b748824c58bc4b36a1b64cd54618a5e4 P 770f563b32324129ae355bd292cfce2e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 280276f5683543e0aeeebad99ea84a84 in term 0.
I20260812 05:20:24.048796 2437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:24.045539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.100.2:47417 (local address 127.31.100.3:37953)
0812 05:20:24.045577 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0812 05:20:24.045579 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 05:20:24.045763 (+ 184us) server_negotiation.cc:407] Connection header received
0812 05:20:24.045805 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:20:24.045806 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:20:24.045815 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:20:24.045860 (+ 45us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 05:20:24.046017 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:24.046500 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:24.047958 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:24.048470 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:24.048533 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:24.048541 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0812 05:20:24.048694 (+ 153us) server_negotiation.cc:299] Negotiation successful
0812 05:20:24.048737 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260812 05:20:24.048974 2323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b748824c58bc4b36a1b64cd54618a5e4" candidate_uuid: "280276f5683543e0aeeebad99ea84a84" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2e7c1aeb46b547e39cddeae55b4bd2a4" is_pre_election: true
I20260812 05:20:24.048949 1989 leader_election.cc:302] T b748824c58bc4b36a1b64cd54618a5e4 P 280276f5683543e0aeeebad99ea84a84 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 280276f5683543e0aeeebad99ea84a84, 770f563b32324129ae355bd292cfce2e; no voters:
I20260812 05:20:24.049049 2323 raft_consensus.cc:2454] T b748824c58bc4b36a1b64cd54618a5e4 P 2e7c1aeb46b547e39cddeae55b4bd2a4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 280276f5683543e0aeeebad99ea84a84 in term 0.
I20260812 05:20:24.049122 2446 raft_consensus.cc:2781] T b748824c58bc4b36a1b64cd54618a5e4 P 280276f5683543e0aeeebad99ea84a84 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 05:20:24.049165 2446 raft_consensus.cc:482] T b748824c58bc4b36a1b64cd54618a5e4 P 280276f5683543e0aeeebad99ea84a84 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 05:20:24.049185 2446 raft_consensus.cc:3037] T b748824c58bc4b36a1b64cd54618a5e4 P 280276f5683543e0aeeebad99ea84a84 [term 0 FOLLOWER]: Advancing to term 1
I20260812 05:20:24.050071 2446 raft_consensus.cc:504] T b748824c58bc4b36a1b64cd54618a5e4 P 280276f5683543e0aeeebad99ea84a84 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "770f563b32324129ae355bd292cfce2e" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 42473 } } peers { permanent_uuid: "280276f5683543e0aeeebad99ea84a84" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 35527 } } peers { permanent_uuid: "2e7c1aeb46b547e39cddeae55b4bd2a4" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 37953 } }
I20260812 05:20:24.050187 2446 leader_election.cc:288] T b748824c58bc4b36a1b64cd54618a5e4 P 280276f5683543e0aeeebad99ea84a84 [CANDIDATE]: Term 1 election: Requested vote from peers 770f563b32324129ae355bd292cfce2e (127.31.100.1:42473), 2e7c1aeb46b547e39cddeae55b4bd2a4 (127.31.100.3:37953)
I20260812 05:20:24.050372 1861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b748824c58bc4b36a1b64cd54618a5e4" candidate_uuid: "280276f5683543e0aeeebad99ea84a84" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "770f563b32324129ae355bd292cfce2e"
I20260812 05:20:24.050412 2323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b748824c58bc4b36a1b64cd54618a5e4" candidate_uuid: "280276f5683543e0aeeebad99ea84a84" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2e7c1aeb46b547e39cddeae55b4bd2a4"
I20260812 05:20:24.050480 2323 raft_consensus.cc:3037] T b748824c58bc4b36a1b64cd54618a5e4 P 2e7c1aeb46b547e39cddeae55b4bd2a4 [term 0 FOLLOWER]: Advancing to term 1
I20260812 05:20:24.050474 1861 raft_consensus.cc:3037] T b748824c58bc4b36a1b64cd54618a5e4 P 770f563b32324129ae355bd292cfce2e [term 0 FOLLOWER]: Advancing to term 1
I20260812 05:20:24.051101 1861 raft_consensus.cc:2454] T b748824c58bc4b36a1b64cd54618a5e4 P 770f563b32324129ae355bd292cfce2e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 280276f5683543e0aeeebad99ea84a84 in term 1.
I20260812 05:20:24.051110 2323 raft_consensus.cc:2454] T b748824c58bc4b36a1b64cd54618a5e4 P 2e7c1aeb46b547e39cddeae55b4bd2a4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 280276f5683543e0aeeebad99ea84a84 in term 1.
I20260812 05:20:24.051242 1989 leader_election.cc:302] T b748824c58bc4b36a1b64cd54618a5e4 P 280276f5683543e0aeeebad99ea84a84 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 280276f5683543e0aeeebad99ea84a84, 2e7c1aeb46b547e39cddeae55b4bd2a4; no voters:
I20260812 05:20:24.051333 2446 raft_consensus.cc:2781] T b748824c58bc4b36a1b64cd54618a5e4 P 280276f5683543e0aeeebad99ea84a84 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 05:20:24.051513 2446 raft_consensus.cc:686] T b748824c58bc4b36a1b64cd54618a5e4 P 280276f5683543e0aeeebad99ea84a84 [term 1 LEADER]: Becoming Leader. State: Replica: 280276f5683543e0aeeebad99ea84a84, State: Running, Role: LEADER
I20260812 05:20:24.051581 2446 consensus_queue.cc:237] T b748824c58bc4b36a1b64cd54618a5e4 P 280276f5683543e0aeeebad99ea84a84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "770f563b32324129ae355bd292cfce2e" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 42473 } } peers { permanent_uuid: "280276f5683543e0aeeebad99ea84a84" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 35527 } } peers { permanent_uuid: "2e7c1aeb46b547e39cddeae55b4bd2a4" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 37953 } }
I20260812 05:20:24.051957 2495 tablet_replica.cc:439] T b748824c58bc4b36a1b64cd54618a5e4 P 280276f5683543e0aeeebad99ea84a84: TxnStatusTablet state changed. Reason: New leader 280276f5683543e0aeeebad99ea84a84. Latest consensus state: current_term: 1 leader_uuid: "280276f5683543e0aeeebad99ea84a84" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "770f563b32324129ae355bd292cfce2e" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 42473 } } peers { permanent_uuid: "280276f5683543e0aeeebad99ea84a84" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 35527 } } peers { permanent_uuid: "2e7c1aeb46b547e39cddeae55b4bd2a4" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 37953 } } }
I20260812 05:20:24.052031 2495 tablet_replica.cc:442] T b748824c58bc4b36a1b64cd54618a5e4 P 280276f5683543e0aeeebad99ea84a84: This TxnStatusTablet replica's current role is: LEADER
I20260812 05:20:24.052184 2497 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260812 05:20:24.052292 2497 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260812 05:20:24.052261 1585 catalog_manager.cc:5495] T b748824c58bc4b36a1b64cd54618a5e4 P 280276f5683543e0aeeebad99ea84a84 reported cstate change: term changed from 0 to 1, leader changed from <none> to 280276f5683543e0aeeebad99ea84a84 (127.31.100.2). New cstate: current_term: 1 leader_uuid: "280276f5683543e0aeeebad99ea84a84" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "770f563b32324129ae355bd292cfce2e" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 42473 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "280276f5683543e0aeeebad99ea84a84" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 35527 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2e7c1aeb46b547e39cddeae55b4bd2a4" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 37953 } health_report { overall_health: UNKNOWN } } }
I20260812 05:20:24.068022 2431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:24.065510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39710 (local address 127.31.100.2:35527)
0812 05:20:24.065552 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0812 05:20:24.065554 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 05:20:24.065594 (+ 40us) server_negotiation.cc:407] Connection header received
0812 05:20:24.065687 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:20:24.065687 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:20:24.065701 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:20:24.065751 (+ 50us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 05:20:24.065864 (+ 113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:24.066304 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:24.067353 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:24.067525 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:24.067570 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:24.067750 (+ 180us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 05:20:24.067833 (+ 83us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 05:20:24.067878 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0812 05:20:24.067959 (+ 81us) server_negotiation.cc:299] Negotiation successful
0812 05:20:24.067976 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260812 05:20:24.068089 2488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:24.065432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.2:35527 (local address 127.0.0.1:39710)
0812 05:20:24.065543 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0812 05:20:24.065554 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0812 05:20:24.065616 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:20:24.065758 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:20:24.065759 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:20:24.065760 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0812 05:20:24.065812 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:20:24.065814 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:20:24.066366 (+ 552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:20:24.066367 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:20:24.067280 (+ 913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:20:24.067283 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:20:24.067589 (+ 306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:20:24.067590 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:20:24.067659 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:24.067676 (+ 17us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0812 05:20:24.067887 (+ 211us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0812 05:20:24.067894 (+ 7us) client_negotiation.cc:769] Sending connection context
0812 05:20:24.067955 (+ 61us) client_negotiation.cc:240] Negotiation successful
0812 05:20:24.068008 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":70}
I20260812 05:20:24.069188 2323 raft_consensus.cc:1261] T b748824c58bc4b36a1b64cd54618a5e4 P 2e7c1aeb46b547e39cddeae55b4bd2a4 [term 1 FOLLOWER]: Refusing update from remote peer 280276f5683543e0aeeebad99ea84a84: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 05:20:24.069195 1861 raft_consensus.cc:1261] T b748824c58bc4b36a1b64cd54618a5e4 P 770f563b32324129ae355bd292cfce2e [term 1 FOLLOWER]: Refusing update from remote peer 280276f5683543e0aeeebad99ea84a84: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 05:20:24.069473 2446 consensus_queue.cc:1035] T b748824c58bc4b36a1b64cd54618a5e4 P 280276f5683543e0aeeebad99ea84a84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "770f563b32324129ae355bd292cfce2e" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 42473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 05:20:24.069603 2495 consensus_queue.cc:1035] T b748824c58bc4b36a1b64cd54618a5e4 P 280276f5683543e0aeeebad99ea84a84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2e7c1aeb46b547e39cddeae55b4bd2a4" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 37953 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 05:20:24.070144 2451 tablet_replica.cc:439] T b748824c58bc4b36a1b64cd54618a5e4 P 2e7c1aeb46b547e39cddeae55b4bd2a4: TxnStatusTablet state changed. Reason: New leader 280276f5683543e0aeeebad99ea84a84. Latest consensus state: current_term: 1 leader_uuid: "280276f5683543e0aeeebad99ea84a84" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "770f563b32324129ae355bd292cfce2e" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 42473 } } peers { permanent_uuid: "280276f5683543e0aeeebad99ea84a84" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 35527 } } peers { permanent_uuid: "2e7c1aeb46b547e39cddeae55b4bd2a4" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 37953 } } }
I20260812 05:20:24.070169 2453 tablet_replica.cc:439] T b748824c58bc4b36a1b64cd54618a5e4 P 770f563b32324129ae355bd292cfce2e: TxnStatusTablet state changed. Reason: New leader 280276f5683543e0aeeebad99ea84a84. Latest consensus state: current_term: 1 leader_uuid: "280276f5683543e0aeeebad99ea84a84" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "770f563b32324129ae355bd292cfce2e" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 42473 } } peers { permanent_uuid: "280276f5683543e0aeeebad99ea84a84" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 35527 } } peers { permanent_uuid: "2e7c1aeb46b547e39cddeae55b4bd2a4" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 37953 } } }
I20260812 05:20:24.070235 2451 tablet_replica.cc:442] T b748824c58bc4b36a1b64cd54618a5e4 P 2e7c1aeb46b547e39cddeae55b4bd2a4: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 05:20:24.070252 2453 tablet_replica.cc:442] T b748824c58bc4b36a1b64cd54618a5e4 P 770f563b32324129ae355bd292cfce2e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 05:20:24.071071 2446 tablet_replica.cc:439] T b748824c58bc4b36a1b64cd54618a5e4 P 280276f5683543e0aeeebad99ea84a84: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "280276f5683543e0aeeebad99ea84a84" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "770f563b32324129ae355bd292cfce2e" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 42473 } } peers { permanent_uuid: "280276f5683543e0aeeebad99ea84a84" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 35527 } } peers { permanent_uuid: "2e7c1aeb46b547e39cddeae55b4bd2a4" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 37953 } } }
I20260812 05:20:24.071147 2446 tablet_replica.cc:442] T b748824c58bc4b36a1b64cd54618a5e4 P 280276f5683543e0aeeebad99ea84a84: This TxnStatusTablet replica's current role is: LEADER
I20260812 05:20:24.071165 2451 tablet_replica.cc:439] T b748824c58bc4b36a1b64cd54618a5e4 P 2e7c1aeb46b547e39cddeae55b4bd2a4: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "280276f5683543e0aeeebad99ea84a84" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "770f563b32324129ae355bd292cfce2e" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 42473 } } peers { permanent_uuid: "280276f5683543e0aeeebad99ea84a84" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 35527 } } peers { permanent_uuid: "2e7c1aeb46b547e39cddeae55b4bd2a4" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 37953 } } }
I20260812 05:20:24.071239 2451 tablet_replica.cc:442] T b748824c58bc4b36a1b64cd54618a5e4 P 2e7c1aeb46b547e39cddeae55b4bd2a4: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 05:20:24.072068 2446 tablet_replica.cc:439] T b748824c58bc4b36a1b64cd54618a5e4 P 280276f5683543e0aeeebad99ea84a84: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "280276f5683543e0aeeebad99ea84a84" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "770f563b32324129ae355bd292cfce2e" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 42473 } } peers { permanent_uuid: "280276f5683543e0aeeebad99ea84a84" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 35527 } } peers { permanent_uuid: "2e7c1aeb46b547e39cddeae55b4bd2a4" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 37953 } } }
I20260812 05:20:24.072149 2446 tablet_replica.cc:442] T b748824c58bc4b36a1b64cd54618a5e4 P 280276f5683543e0aeeebad99ea84a84: This TxnStatusTablet replica's current role is: LEADER
I20260812 05:20:24.072198 2456 mvcc.cc:205] Tried to move back new op lower bound from 7317553250585395200 to 7317553250515750912. Current Snapshot: MvccSnapshot[applied={T|T < 7317553250585395200}]
I20260812 05:20:24.072439 2453 tablet_replica.cc:439] T b748824c58bc4b36a1b64cd54618a5e4 P 770f563b32324129ae355bd292cfce2e: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "280276f5683543e0aeeebad99ea84a84" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "770f563b32324129ae355bd292cfce2e" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 42473 } } peers { permanent_uuid: "280276f5683543e0aeeebad99ea84a84" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 35527 } } peers { permanent_uuid: "2e7c1aeb46b547e39cddeae55b4bd2a4" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 37953 } } }
I20260812 05:20:24.072512 2453 tablet_replica.cc:442] T b748824c58bc4b36a1b64cd54618a5e4 P 770f563b32324129ae355bd292cfce2e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 05:20:24.077282 2431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:24.074867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.100.1:42283 (local address 127.31.100.2:35527)
0812 05:20:24.074905 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0812 05:20:24.074907 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 05:20:24.074983 (+ 76us) server_negotiation.cc:407] Connection header received
0812 05:20:24.075057 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:20:24.075057 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:20:24.075065 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:20:24.075104 (+ 39us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 05:20:24.075211 (+ 107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:24.075629 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:24.076700 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:24.076885 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:24.076945 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:24.077108 (+ 163us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 05:20:24.077134 (+ 26us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 05:20:24.077169 (+ 35us) server_negotiation.cc:1035] Waiting for connection context
0812 05:20:24.077221 (+ 52us) server_negotiation.cc:299] Negotiation successful
0812 05:20:24.077238 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260812 05:20:24.077343 2510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:24.074776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.2:35527 (local address 127.31.100.1:42283)
0812 05:20:24.074928 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0812 05:20:24.074939 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0812 05:20:24.074999 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:20:24.075109 (+ 110us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:20:24.075109 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:20:24.075111 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0812 05:20:24.075158 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:20:24.075160 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:20:24.075688 (+ 528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:20:24.075688 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:20:24.076603 (+ 915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:20:24.076606 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:20:24.076956 (+ 350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:20:24.076956 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:20:24.077015 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:24.077031 (+ 16us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0812 05:20:24.077172 (+ 141us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0812 05:20:24.077173 (+ 1us) client_negotiation.cc:769] Sending connection context
0812 05:20:24.077219 (+ 46us) client_negotiation.cc:240] Negotiation successful
0812 05:20:24.077264 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":78,"threads_started":1}
I20260812 05:20:24.079625 2323 raft_consensus.cc:1261] T 2ae73929604d484583fac8ed20620889 P 2e7c1aeb46b547e39cddeae55b4bd2a4 [term 1 FOLLOWER]: Refusing update from remote peer 770f563b32324129ae355bd292cfce2e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 05:20:24.079651 2103 raft_consensus.cc:1261] T 2ae73929604d484583fac8ed20620889 P 280276f5683543e0aeeebad99ea84a84 [term 1 FOLLOWER]: Refusing update from remote peer 770f563b32324129ae355bd292cfce2e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 05:20:24.079808 2453 consensus_queue.cc:1035] T 2ae73929604d484583fac8ed20620889 P 770f563b32324129ae355bd292cfce2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "280276f5683543e0aeeebad99ea84a84" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 35527 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 05:20:24.079924 2447 consensus_queue.cc:1035] T 2ae73929604d484583fac8ed20620889 P 770f563b32324129ae355bd292cfce2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "2e7c1aeb46b547e39cddeae55b4bd2a4" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 37953 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 05:20:24.082535 2458 mvcc.cc:205] Tried to move back new op lower bound from 7317553250628886528 to 7317553250110304256. Current Snapshot: MvccSnapshot[applied={T|T < 7317553250628886528}]
I20260812 05:20:24.088002 2185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:24.085357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.1:42473 (local address 127.31.100.2:39771)
0812 05:20:24.085495 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0812 05:20:24.085504 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0812 05:20:24.085540 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:20:24.085638 (+ 98us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:20:24.085638 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:20:24.085639 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0812 05:20:24.085691 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:20:24.085692 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:20:24.086274 (+ 582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:20:24.086275 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:20:24.087219 (+ 944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:20:24.087222 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:20:24.087677 (+ 455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:20:24.087678 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:20:24.087733 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:24.087746 (+ 13us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0812 05:20:24.087900 (+ 154us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0812 05:20:24.087901 (+ 1us) client_negotiation.cc:769] Sending connection context
0812 05:20:24.087925 (+ 24us) client_negotiation.cc:240] Negotiation successful
0812 05:20:24.087956 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":76}
I20260812 05:20:24.088027 2434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:24.085481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.100.2:39771 (local address 127.31.100.1:42473)
0812 05:20:24.085518 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0812 05:20:24.085520 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 05:20:24.085531 (+ 11us) server_negotiation.cc:407] Connection header received
0812 05:20:24.085586 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:20:24.085586 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:20:24.085594 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:20:24.085634 (+ 40us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 05:20:24.085741 (+ 107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:24.086185 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:24.087316 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:24.087610 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:24.087673 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:24.087815 (+ 142us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 05:20:24.087848 (+ 33us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 05:20:24.087899 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0812 05:20:24.087941 (+ 42us) server_negotiation.cc:299] Negotiation successful
0812 05:20:24.087961 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260812 05:20:24.096522 1585 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34604:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 05:20:24.096613 1585 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34604:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 05:20:24.097188 1585 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3d208a9824f844afb0da31ac8bb9bc3a: Sending DeleteTablet for 3 replicas of tablet 2ae73929604d484583fac8ed20620889
I20260812 05:20:24.097463 2073 tablet_service.cc:1514] Processing DeleteTablet for tablet 2ae73929604d484583fac8ed20620889 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 05:20:24 UTC) from {username='kudu', principal='kudu/127.31.100.62@KRBTEST.COM'} at 127.0.0.1:39704
I20260812 05:20:24.097466 1827 tablet_service.cc:1514] Processing DeleteTablet for tablet 2ae73929604d484583fac8ed20620889 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 05:20:24 UTC) from {username='kudu', principal='kudu/127.31.100.62@KRBTEST.COM'} at 127.0.0.1:38296
I20260812 05:20:24.097555 2471 tablet_replica.cc:330] T 2ae73929604d484583fac8ed20620889 P 770f563b32324129ae355bd292cfce2e: stopping tablet replica
I20260812 05:20:24.097563 2472 tablet_replica.cc:330] T 2ae73929604d484583fac8ed20620889 P 280276f5683543e0aeeebad99ea84a84: stopping tablet replica
I20260812 05:20:24.097606 2471 raft_consensus.cc:2229] T 2ae73929604d484583fac8ed20620889 P 770f563b32324129ae355bd292cfce2e [term 1 LEADER]: Raft consensus shutting down.
I20260812 05:20:24.097608 2472 raft_consensus.cc:2229] T 2ae73929604d484583fac8ed20620889 P 280276f5683543e0aeeebad99ea84a84 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 05:20:24.097645 2472 raft_consensus.cc:2258] T 2ae73929604d484583fac8ed20620889 P 280276f5683543e0aeeebad99ea84a84 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 05:20:24.097659 2471 raft_consensus.cc:2258] T 2ae73929604d484583fac8ed20620889 P 770f563b32324129ae355bd292cfce2e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 05:20:24.097944 2472 ts_tablet_manager.cc:1905] T 2ae73929604d484583fac8ed20620889 P 280276f5683543e0aeeebad99ea84a84: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 05:20:24.098028 2471 ts_tablet_manager.cc:1905] T 2ae73929604d484583fac8ed20620889 P 770f563b32324129ae355bd292cfce2e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 05:20:24.099498 2471 ts_tablet_manager.cc:1918] T 2ae73929604d484583fac8ed20620889 P 770f563b32324129ae355bd292cfce2e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260812 05:20:24.099560 2471 log.cc:1196] T 2ae73929604d484583fac8ed20620889 P 770f563b32324129ae355bd292cfce2e: Deleting WAL directory at /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786512017613366-32144-0/minicluster-data/ts-0/wal/wals/2ae73929604d484583fac8ed20620889
I20260812 05:20:24.099630 2472 ts_tablet_manager.cc:1918] T 2ae73929604d484583fac8ed20620889 P 280276f5683543e0aeeebad99ea84a84: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260812 05:20:24.099689 2472 log.cc:1196] T 2ae73929604d484583fac8ed20620889 P 280276f5683543e0aeeebad99ea84a84: Deleting WAL directory at /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786512017613366-32144-0/minicluster-data/ts-1/wal/wals/2ae73929604d484583fac8ed20620889
I20260812 05:20:24.099756 2471 ts_tablet_manager.cc:1939] T 2ae73929604d484583fac8ed20620889 P 770f563b32324129ae355bd292cfce2e: Deleting consensus metadata
I20260812 05:20:24.099915 2472 ts_tablet_manager.cc:1939] T 2ae73929604d484583fac8ed20620889 P 280276f5683543e0aeeebad99ea84a84: Deleting consensus metadata
I20260812 05:20:24.100041 1564 catalog_manager.cc:4841] TS 770f563b32324129ae355bd292cfce2e (127.31.100.1:42473): tablet 2ae73929604d484583fac8ed20620889 (table test-table [id=1007f7c68ded45bba9685132b892472c]) successfully deleted
I20260812 05:20:24.100123 2287 tablet_service.cc:1514] Processing DeleteTablet for tablet 2ae73929604d484583fac8ed20620889 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 05:20:24 UTC) from {username='kudu', principal='kudu/127.31.100.62@KRBTEST.COM'} at 127.0.0.1:41672
I20260812 05:20:24.100234 2474 tablet_replica.cc:330] T 2ae73929604d484583fac8ed20620889 P 2e7c1aeb46b547e39cddeae55b4bd2a4: stopping tablet replica
I20260812 05:20:24.100296 2474 raft_consensus.cc:2229] T 2ae73929604d484583fac8ed20620889 P 2e7c1aeb46b547e39cddeae55b4bd2a4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 05:20:24.100335 2474 raft_consensus.cc:2258] T 2ae73929604d484583fac8ed20620889 P 2e7c1aeb46b547e39cddeae55b4bd2a4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 05:20:24.100415 1564 catalog_manager.cc:4841] TS 280276f5683543e0aeeebad99ea84a84 (127.31.100.2:35527): tablet 2ae73929604d484583fac8ed20620889 (table test-table [id=1007f7c68ded45bba9685132b892472c]) successfully deleted
I20260812 05:20:24.100634 2474 ts_tablet_manager.cc:1905] T 2ae73929604d484583fac8ed20620889 P 2e7c1aeb46b547e39cddeae55b4bd2a4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 05:20:24.101904 2474 ts_tablet_manager.cc:1918] T 2ae73929604d484583fac8ed20620889 P 2e7c1aeb46b547e39cddeae55b4bd2a4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260812 05:20:24.101956 2474 log.cc:1196] T 2ae73929604d484583fac8ed20620889 P 2e7c1aeb46b547e39cddeae55b4bd2a4: Deleting WAL directory at /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786512017613366-32144-0/minicluster-data/ts-2/wal/wals/2ae73929604d484583fac8ed20620889
I20260812 05:20:24.102125 2474 ts_tablet_manager.cc:1939] T 2ae73929604d484583fac8ed20620889 P 2e7c1aeb46b547e39cddeae55b4bd2a4: Deleting consensus metadata
I20260812 05:20:24.102413 1564 catalog_manager.cc:4841] TS 2e7c1aeb46b547e39cddeae55b4bd2a4 (127.31.100.3:37953): tablet 2ae73929604d484583fac8ed20620889 (table test-table [id=1007f7c68ded45bba9685132b892472c]) successfully deleted
Aug 12 05:20:24 dist-test-slave-42z9 krb5kdc[1515](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512023, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.31.100.1@KRBTEST.COM
I20260812 05:20:24.107446 2434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:24.098171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38320 (local address 127.31.100.1:42473)
0812 05:20:24.098209 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0812 05:20:24.098211 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 05:20:24.098441 (+ 230us) server_negotiation.cc:407] Connection header received
0812 05:20:24.098504 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:20:24.098505 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:20:24.098516 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:20:24.098558 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:20:24.099283 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:24.099982 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:24.101123 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:24.101386 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:24.101501 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:24.104881 (+ 3380us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 05:20:24.104889 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 05:20:24.104889 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 05:20:24.104895 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0812 05:20:24.106414 (+ 1519us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:20:24.106780 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:20:24.106782 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:20:24.106782 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:20:24.106832 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:20:24.107006 (+ 174us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:20:24.107007 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:20:24.107007 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:20:24.107120 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 05:20:24.107216 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0812 05:20:24.107328 (+ 112us) server_negotiation.cc:299] Negotiation successful
0812 05:20:24.107390 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
W20260812 05:20:24.107895 1934 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38320
I20260812 05:20:24.114921 1676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:24.108757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34620 (local address 127.31.100.62:35317)
0812 05:20:24.108798 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0812 05:20:24.108800 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 05:20:24.108844 (+ 44us) server_negotiation.cc:407] Connection header received
0812 05:20:24.108911 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:20:24.108912 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:20:24.108920 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:20:24.108961 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:20:24.109518 (+ 557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:24.109988 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:24.110957 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:24.111141 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:24.111208 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:24.112422 (+ 1214us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 05:20:24.112423 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 05:20:24.112423 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 05:20:24.112429 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0812 05:20:24.113771 (+ 1342us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:20:24.114167 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:20:24.114169 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:20:24.114169 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:20:24.114244 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:20:24.114480 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:20:24.114481 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:20:24.114481 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:20:24.114595 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 05:20:24.114675 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0812 05:20:24.114794 (+ 119us) server_negotiation.cc:299] Negotiation successful
0812 05:20:24.114863 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
W20260812 05:20:24.115306 1585 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34620
I20260812 05:20:24.115988 32144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0waFvT/build/release/bin/kudu with pid 1712
I20260812 05:20:24.130230 32144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0waFvT/build/release/bin/kudu with pid 1966
I20260812 05:20:24.142633 32144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0waFvT/build/release/bin/kudu with pid 2191
I20260812 05:20:24.152549 32144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0waFvT/build/release/bin/kudu with pid 1538
2026-08-12T05:20:24Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (2204 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786512017613366-32144-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 05:20:24 dist-test-slave-42z9 krb5kdc[2531](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 05:20:24 dist-test-slave-42z9 krb5kdc[2531](info): set up 2 sockets
Aug 12 05:20:24 dist-test-slave-42z9 krb5kdc[2531](info): commencing operation
krb5kdc: starting...
W20260812 05:20:25.316521 32144 mini_kdc.cc:121] Time spent starting KDC: real 1.139s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786512017613366-32144-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786512017613366-32144-0/krb5kdc/test-user.keytab.
Aug 12 05:20:25 dist-test-slave-42z9 krb5kdc[2531](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512025, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T05:20:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T05:20:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.100.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.100.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.100.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.62.keytab.
Entry for principal kudu/127.31.100.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.62.keytab.
I20260812 05:20:25.427786 32144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0waFvT/build/release/bin/kudu
/tmp/dist-test-task0waFvT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786512017613366-32144-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786512017613366-32144-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786512017613366-32144-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786512017613366-32144-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.100.62:45993
--webserver_interface=127.31.100.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.62.keytab
--principal=kudu/127.31.100.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.100.20:34405
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.100.62:45993
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786512017613366-32144-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786512017613366-32144-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786512017613366-32144-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 05:20:25.507568 2552 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 05:20:25.507771 2552 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 05:20:25.507800 2552 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 05:20:25.508908 2552 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 05:20:25.508947 2552 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260812 05:20:25.508965 2552 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 05:20:25.508982 2552 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 05:20:25.508997 2552 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 05:20:25.509014 2552 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 05:20:25.510473 2552 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.100.20:34405
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786512017613366-32144-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786512017613366-32144-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.100.62:45993
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.62.keytab
--principal=kudu/127.31.100.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.100.62:45993
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786512017613366-32144-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.100.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2552
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786512017613366-32144-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 04:20:34 UTC on ccycloud.quasar-phqdqp.root.comops.site
build id 4581871
I20260812 05:20:25.510716 2552 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 05:20:25.510913 2552 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 05:20:25.515538 2566 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 05:20:25.515563 2561 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 05:20:25.515740 2563 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 05:20:25.515789 2552 server_base.cc:1034] running on GCE node
I20260812 05:20:25.516016 2552 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 05:20:25.516290 2552 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 05:20:25.517464 2552 hybrid_clock.cc:648] HybridClock initialized: now 1786512025517410 us; error 28 us; skew 500 ppm
Aug 12 05:20:25 dist-test-slave-42z9 krb5kdc[2531](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512025, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.100.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 05:20:25.519829 2552 init.cc:377] Logged in from keytab as kudu/127.31.100.62@KRBTEST.COM (short username kudu)
I20260812 05:20:25.520282 2552 webserver.cc:460] Webserver started at http://127.31.100.62:33881/ using document root <none> and password file <none>
I20260812 05:20:25.520500 2552 fs_manager.cc:359] Metadata directory not provided
I20260812 05:20:25.520552 2552 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 05:20:25.520675 2552 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 05:20:25.521739 2552 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786512017613366-32144-0/minicluster-data/master-0/data/instance:
uuid: "47f84840accd4b84836532f25b4a420e"
format_stamp: "Formatted at 2026-08-12 05:20:25 on dist-test-slave-42z9"
server_key: "4bc4eb61fb10e7399c77da2c8dec76fb"
server_key_iv: "4eefe2dea20c1e3dcb2b19dcbbc54627"
server_key_version: "encryptionkey@0"
I20260812 05:20:25.522063 2552 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786512017613366-32144-0/minicluster-data/master-0/wal/instance:
uuid: "47f84840accd4b84836532f25b4a420e"
format_stamp: "Formatted at 2026-08-12 05:20:25 on dist-test-slave-42z9"
server_key: "4bc4eb61fb10e7399c77da2c8dec76fb"
server_key_iv: "4eefe2dea20c1e3dcb2b19dcbbc54627"
server_key_version: "encryptionkey@0"
I20260812 05:20:25.523298 2552 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260812 05:20:25.523995 2577 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 05:20:25.524158 2552 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260812 05:20:25.524245 2552 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786512017613366-32144-0/minicluster-data/master-0/data,/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786512017613366-32144-0/minicluster-data/master-0/wal
uuid: "47f84840accd4b84836532f25b4a420e"
format_stamp: "Formatted at 2026-08-12 05:20:25 on dist-test-slave-42z9"
server_key: "4bc4eb61fb10e7399c77da2c8dec76fb"
server_key_iv: "4eefe2dea20c1e3dcb2b19dcbbc54627"
server_key_version: "encryptionkey@0"
I20260812 05:20:25.524304 2552 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786512017613366-32144-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786512017613366-32144-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786512017613366-32144-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 05:20:25.533020 2552 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 05:20:25.534858 2552 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 05:20:25.534986 2552 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 05:20:25.539542 2552 rpc_server.cc:307] RPC server started. Bound to: 127.31.100.62:45993
I20260812 05:20:25.539605 2682 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.100.62:45993 every 8 connection(s)
I20260812 05:20:25.539896 2552 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786512017613366-32144-0/minicluster-data/master-0/data/info.pb
I20260812 05:20:25.540518 2683 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 05:20:25.541815 32144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0waFvT/build/release/bin/kudu as pid 2552
I20260812 05:20:25.541891 32144 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786512017613366-32144-0/minicluster-data/master-0/wal/instance
I20260812 05:20:25.542039 32144 external_mini_cluster.cc:1428] Setting key 61eec14bd13acd13b65df006a7c65cd1
I20260812 05:20:25.543217 2683 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 47f84840accd4b84836532f25b4a420e: Bootstrap starting.
I20260812 05:20:25.543896 2683 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 47f84840accd4b84836532f25b4a420e: Neither blocks nor log segments found. Creating new log.
I20260812 05:20:25.544166 2683 log.cc:824] T 00000000000000000000000000000000 P 47f84840accd4b84836532f25b4a420e: Log is configured to *not* fsync() on all Append() calls
I20260812 05:20:25.544819 2683 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 47f84840accd4b84836532f25b4a420e: No bootstrap required, opened a new log
I20260812 05:20:25.546718 2683 raft_consensus.cc:348] T 00000000000000000000000000000000 P 47f84840accd4b84836532f25b4a420e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47f84840accd4b84836532f25b4a420e" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 45993 } }
I20260812 05:20:25.546895 2683 raft_consensus.cc:374] T 00000000000000000000000000000000 P 47f84840accd4b84836532f25b4a420e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 05:20:25.546948 2683 raft_consensus.cc:729] T 00000000000000000000000000000000 P 47f84840accd4b84836532f25b4a420e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47f84840accd4b84836532f25b4a420e, State: Initialized, Role: FOLLOWER
I20260812 05:20:25.547077 2683 consensus_queue.cc:260] T 00000000000000000000000000000000 P 47f84840accd4b84836532f25b4a420e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47f84840accd4b84836532f25b4a420e" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 45993 } }
I20260812 05:20:25.547150 2683 raft_consensus.cc:388] T 00000000000000000000000000000000 P 47f84840accd4b84836532f25b4a420e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 05:20:25.547183 2683 raft_consensus.cc:482] T 00000000000000000000000000000000 P 47f84840accd4b84836532f25b4a420e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 05:20:25.547228 2683 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 47f84840accd4b84836532f25b4a420e [term 0 FOLLOWER]: Advancing to term 1
I20260812 05:20:25.547722 2683 raft_consensus.cc:504] T 00000000000000000000000000000000 P 47f84840accd4b84836532f25b4a420e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47f84840accd4b84836532f25b4a420e" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 45993 } }
I20260812 05:20:25.547823 2683 leader_election.cc:302] T 00000000000000000000000000000000 P 47f84840accd4b84836532f25b4a420e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 47f84840accd4b84836532f25b4a420e; no voters:
I20260812 05:20:25.547983 2683 leader_election.cc:288] T 00000000000000000000000000000000 P 47f84840accd4b84836532f25b4a420e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 05:20:25.548012 2691 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 47f84840accd4b84836532f25b4a420e [term 1 FOLLOWER]: Leader election won for term 1
I20260812 05:20:25.548130 2691 raft_consensus.cc:686] T 00000000000000000000000000000000 P 47f84840accd4b84836532f25b4a420e [term 1 LEADER]: Becoming Leader. State: Replica: 47f84840accd4b84836532f25b4a420e, State: Running, Role: LEADER
Aug 12 05:20:25 dist-test-slave-42z9 krb5kdc[2531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512025, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.100.62@KRBTEST.COM
I20260812 05:20:25.548240 2683 sys_catalog.cc:564] T 00000000000000000000000000000000 P 47f84840accd4b84836532f25b4a420e [sys.catalog]: configured and running, proceeding with master startup.
I20260812 05:20:25.548223 2691 consensus_queue.cc:237] T 00000000000000000000000000000000 P 47f84840accd4b84836532f25b4a420e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47f84840accd4b84836532f25b4a420e" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 45993 } }
I20260812 05:20:25.548568 2691 sys_catalog.cc:455] T 00000000000000000000000000000000 P 47f84840accd4b84836532f25b4a420e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 47f84840accd4b84836532f25b4a420e. Latest consensus state: current_term: 1 leader_uuid: "47f84840accd4b84836532f25b4a420e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47f84840accd4b84836532f25b4a420e" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 45993 } } }
I20260812 05:20:25.548648 2691 sys_catalog.cc:458] T 00000000000000000000000000000000 P 47f84840accd4b84836532f25b4a420e [sys.catalog]: This master's current role is: LEADER
I20260812 05:20:25.548586 2693 sys_catalog.cc:455] T 00000000000000000000000000000000 P 47f84840accd4b84836532f25b4a420e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "47f84840accd4b84836532f25b4a420e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47f84840accd4b84836532f25b4a420e" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 45993 } } }
I20260812 05:20:25.548820 2693 sys_catalog.cc:458] T 00000000000000000000000000000000 P 47f84840accd4b84836532f25b4a420e [sys.catalog]: This master's current role is: LEADER
I20260812 05:20:25.549022 2698 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 05:20:25.549561 2698 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 05:20:25.552155 2688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:25.542533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44238 (local address 127.31.100.62:45993)
0812 05:20:25.542741 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0812 05:20:25.542752 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0812 05:20:25.542788 (+ 36us) server_negotiation.cc:407] Connection header received
0812 05:20:25.543285 (+ 497us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:20:25.543294 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:20:25.543371 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:20:25.543514 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:20:25.544060 (+ 546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:25.545006 (+ 946us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:25.545784 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:25.546194 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:25.546288 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:25.548732 (+ 2444us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 05:20:25.548736 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 05:20:25.548736 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 05:20:25.548748 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0812 05:20:25.550621 (+ 1873us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:20:25.550985 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:20:25.550987 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:20:25.550987 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:20:25.551074 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:20:25.551258 (+ 184us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:20:25.551258 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:20:25.551258 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:20:25.551504 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 05:20:25.551606 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0812 05:20:25.551738 (+ 132us) server_negotiation.cc:299] Negotiation successful
0812 05:20:25.551878 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153,"thread_start_us":87,"threads_started":1}
I20260812 05:20:25.553192 2698 catalog_manager.cc:1347] Generated new cluster ID: 93ec49d64b044adbba4d2b4709d78d55
I20260812 05:20:25.553251 2698 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 05:20:25.566813 2698 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 05:20:25.567287 2698 catalog_manager.cc:1504] Loading token signing keys...
I20260812 05:20:25.573228 2698 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 47f84840accd4b84836532f25b4a420e: Generated new TSK 0
I20260812 05:20:25.573382 2698 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.31.100.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.100.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.100.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.1.keytab.
Entry for principal kudu/127.31.100.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.1.keytab.
I20260812 05:20:25.602243 32144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0waFvT/build/release/bin/kudu
/tmp/dist-test-task0waFvT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786512017613366-32144-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786512017613366-32144-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786512017613366-32144-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786512017613366-32144-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.100.1:0
--local_ip_for_outbound_sockets=127.31.100.1
--webserver_interface=127.31.100.1
--webserver_port=0
--tserver_master_addrs=127.31.100.62:45993
--keytab_file=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.1.keytab
--principal=kudu/127.31.100.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.100.20:34405
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786512017613366-32144-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786512017613366-32144-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786512017613366-32144-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 05:20:25.683012 2719 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 05:20:25.683200 2719 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 05:20:25.683233 2719 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 05:20:25.684316 2719 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 05:20:25.684353 2719 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 05:20:25.684379 2719 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260812 05:20:25.684394 2719 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260812 05:20:25.684427 2719 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.100.1
I20260812 05:20:25.685881 2719 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.100.20:34405
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786512017613366-32144-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786512017613366-32144-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.1.keytab
--principal=kudu/127.31.100.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.100.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786512017613366-32144-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.100.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.31.100.62:45993
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2719
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.100.1
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786512017613366-32144-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 04:20:34 UTC on ccycloud.quasar-phqdqp.root.comops.site
build id 4581871
I20260812 05:20:25.686118 2719 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 05:20:25.686311 2719 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 05:20:25.690933 2734 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 05:20:25.691119 2726 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 05:20:25.690949 2727 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 05:20:25.691169 2719 server_base.cc:1034] running on GCE node
I20260812 05:20:25.691371 2719 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 05:20:25.691589 2719 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 05:20:25.692735 2719 hybrid_clock.cc:648] HybridClock initialized: now 1786512025692723 us; error 29 us; skew 500 ppm
Aug 12 05:20:25 dist-test-slave-42z9 krb5kdc[2531](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512025, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.100.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 05:20:25.695062 2719 init.cc:377] Logged in from keytab as kudu/127.31.100.1@KRBTEST.COM (short username kudu)
I20260812 05:20:25.695429 2719 webserver.cc:460] Webserver started at http://127.31.100.1:38605/ using document root <none> and password file <none>
I20260812 05:20:25.695621 2719 fs_manager.cc:359] Metadata directory not provided
I20260812 05:20:25.695658 2719 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 05:20:25.695748 2719 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 05:20:25.696713 2719 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786512017613366-32144-0/minicluster-data/ts-0/data/instance:
uuid: "4dce95e2db6d454f9a636ef4f25cd23b"
format_stamp: "Formatted at 2026-08-12 05:20:25 on dist-test-slave-42z9"
server_key: "07386663dab46a9055389e5feac624d6"
server_key_iv: "f3619010c5115e49e6b68ab72c3593f7"
server_key_version: "encryptionkey@0"
I20260812 05:20:25.696998 2719 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786512017613366-32144-0/minicluster-data/ts-0/wal/instance:
uuid: "4dce95e2db6d454f9a636ef4f25cd23b"
format_stamp: "Formatted at 2026-08-12 05:20:25 on dist-test-slave-42z9"
server_key: "07386663dab46a9055389e5feac624d6"
server_key_iv: "f3619010c5115e49e6b68ab72c3593f7"
server_key_version: "encryptionkey@0"
I20260812 05:20:25.698235 2719 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260812 05:20:25.698880 2755 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 05:20:25.699030 2719 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260812 05:20:25.699112 2719 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786512017613366-32144-0/minicluster-data/ts-0/data,/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786512017613366-32144-0/minicluster-data/ts-0/wal
uuid: "4dce95e2db6d454f9a636ef4f25cd23b"
format_stamp: "Formatted at 2026-08-12 05:20:25 on dist-test-slave-42z9"
server_key: "07386663dab46a9055389e5feac624d6"
server_key_iv: "f3619010c5115e49e6b68ab72c3593f7"
server_key_version: "encryptionkey@0"
I20260812 05:20:25.699167 2719 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786512017613366-32144-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786512017613366-32144-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786512017613366-32144-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 05:20:25.720666 2719 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 05:20:25.722267 2719 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 05:20:25.722366 2719 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 05:20:25.722779 2719 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 05:20:25.722818 2719 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 05:20:25.722836 2719 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 05:20:25.722847 2719 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 05:20:25.728936 2719 rpc_server.cc:307] RPC server started. Bound to: 127.31.100.1:46417
I20260812 05:20:25.729017 2958 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.100.1:46417 every 8 connection(s)
Aug 12 05:20:25 dist-test-slave-42z9 krb5kdc[2531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512025, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.100.1@KRBTEST.COM for kudu/127.31.100.62@KRBTEST.COM
I20260812 05:20:25.729270 2719 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786512017613366-32144-0/minicluster-data/ts-0/data/info.pb
I20260812 05:20:25.731776 2688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:25.723196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.100.1:42723 (local address 127.31.100.62:45993)
0812 05:20:25.723261 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0812 05:20:25.723263 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 05:20:25.723412 (+ 149us) server_negotiation.cc:407] Connection header received
0812 05:20:25.723835 (+ 423us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:20:25.723836 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:20:25.723848 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:20:25.723918 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:20:25.724684 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:25.725434 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:25.726913 (+ 1479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:25.727222 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:25.727305 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:25.729733 (+ 2428us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 05:20:25.729734 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 05:20:25.729735 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 05:20:25.729741 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0812 05:20:25.730785 (+ 1044us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:20:25.731133 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:20:25.731133 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:20:25.731134 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:20:25.731176 (+ 42us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:20:25.731342 (+ 166us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:20:25.731343 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:20:25.731343 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:20:25.731430 (+ 87us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 05:20:25.731471 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0812 05:20:25.731651 (+ 180us) server_negotiation.cc:299] Negotiation successful
0812 05:20:25.731714 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260812 05:20:25.731925 2777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:25.723136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.62:45993 (local address 127.31.100.1:42723)
0812 05:20:25.723289 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0812 05:20:25.723326 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0812 05:20:25.723748 (+ 422us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:20:25.723982 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:20:25.723986 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:20:25.724282 (+ 296us) client_negotiation.cc:189] Negotiated authn=SASL
0812 05:20:25.724584 (+ 302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:20:25.724590 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:20:25.725537 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:20:25.725538 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:20:25.726787 (+ 1249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:20:25.726790 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:20:25.727307 (+ 517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:20:25.727308 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:20:25.727405 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:25.727903 (+ 498us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 05:20:25.727912 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0812 05:20:25.729651 (+ 1739us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 05:20:25.730852 (+ 1201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 05:20:25.730859 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 05:20:25.730860 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 05:20:25.731080 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 05:20:25.731219 (+ 139us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 05:20:25.731220 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 05:20:25.731220 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 05:20:25.731302 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 05:20:25.731471 (+ 169us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 05:20:25.731473 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 05:20:25.731569 (+ 96us) client_negotiation.cc:769] Sending connection context
0812 05:20:25.731640 (+ 71us) client_negotiation.cc:240] Negotiation successful
0812 05:20:25.731746 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":84,"threads_started":1}
I20260812 05:20:25.732383 2959 heartbeater.cc:347] Connected to a master server at 127.31.100.62:45993
I20260812 05:20:25.732465 2959 heartbeater.cc:464] Registering TS with master...
I20260812 05:20:25.732620 2959 heartbeater.cc:511] Master 127.31.100.62:45993 requested a full tablet report, sending...
I20260812 05:20:25.733069 2606 ts_manager.cc:194] Registered new tserver with Master: 4dce95e2db6d454f9a636ef4f25cd23b (127.31.100.1:46417)
I20260812 05:20:25.733781 2606 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.100.1@KRBTEST.COM'} at 127.31.100.1:42723
I20260812 05:20:25.736765 32144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0waFvT/build/release/bin/kudu as pid 2719
I20260812 05:20:25.736913 32144 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786512017613366-32144-0/minicluster-data/ts-0/wal/instance
I20260812 05:20:25.737098 32144 external_mini_cluster.cc:1428] Setting key 2d124c49f09e40ba7f12b475c0ec0efc
I20260812 05:20:25.738255 2967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:25.733284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.62:45993 (local address 127.31.100.1:59533)
0812 05:20:25.733414 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0812 05:20:25.733424 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0812 05:20:25.733501 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:20:25.733618 (+ 117us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:20:25.733619 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:20:25.733766 (+ 147us) client_negotiation.cc:189] Negotiated authn=SASL
0812 05:20:25.733816 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:20:25.733817 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:20:25.734566 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:20:25.734567 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:20:25.735061 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:20:25.735062 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:20:25.735337 (+ 275us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:20:25.735337 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:20:25.735387 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:25.735678 (+ 291us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 05:20:25.735682 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0812 05:20:25.735941 (+ 259us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 05:20:25.737322 (+ 1381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 05:20:25.737323 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 05:20:25.737323 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 05:20:25.737530 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 05:20:25.737699 (+ 169us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 05:20:25.737700 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 05:20:25.737700 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 05:20:25.737757 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 05:20:25.738025 (+ 268us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 05:20:25.738025 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 05:20:25.738060 (+ 35us) client_negotiation.cc:769] Sending connection context
0812 05:20:25.738119 (+ 59us) client_negotiation.cc:240] Negotiation successful
0812 05:20:25.738190 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":80,"threads_started":1}
I20260812 05:20:25.738297 2688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:25.733329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.100.1:59533 (local address 127.31.100.62:45993)
0812 05:20:25.733362 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0812 05:20:25.733364 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 05:20:25.733487 (+ 123us) server_negotiation.cc:407] Connection header received
0812 05:20:25.733552 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:20:25.733553 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:20:25.733562 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:20:25.733605 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:20:25.733881 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:25.734484 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:25.735120 (+ 636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:25.735289 (+ 169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:25.735332 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:25.736028 (+ 696us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 05:20:25.736030 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 05:20:25.736030 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 05:20:25.736036 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0812 05:20:25.737251 (+ 1215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:20:25.737590 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:20:25.737591 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:20:25.737591 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:20:25.737646 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:20:25.737820 (+ 174us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:20:25.737820 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:20:25.737821 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:20:25.737949 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 05:20:25.738023 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0812 05:20:25.738157 (+ 134us) server_negotiation.cc:299] Negotiation successful
0812 05:20:25.738226 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
WARNING: no policy specified for kudu/127.31.100.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.100.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.100.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.2.keytab.
Entry for principal kudu/127.31.100.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.2.keytab.
I20260812 05:20:25.760174 32144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0waFvT/build/release/bin/kudu
/tmp/dist-test-task0waFvT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786512017613366-32144-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786512017613366-32144-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786512017613366-32144-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786512017613366-32144-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.100.2:0
--local_ip_for_outbound_sockets=127.31.100.2
--webserver_interface=127.31.100.2
--webserver_port=0
--tserver_master_addrs=127.31.100.62:45993
--keytab_file=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.2.keytab
--principal=kudu/127.31.100.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.100.20:34405
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786512017613366-32144-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786512017613366-32144-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786512017613366-32144-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 05:20:25.839862 2980 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 05:20:25.840054 2980 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 05:20:25.840085 2980 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 05:20:25.841176 2980 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 05:20:25.841214 2980 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 05:20:25.841241 2980 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260812 05:20:25.841257 2980 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260812 05:20:25.841287 2980 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.100.2
I20260812 05:20:25.842726 2980 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.100.20:34405
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786512017613366-32144-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786512017613366-32144-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.2.keytab
--principal=kudu/127.31.100.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.100.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786512017613366-32144-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.100.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.31.100.62:45993
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2980
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.100.2
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786512017613366-32144-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 04:20:34 UTC on ccycloud.quasar-phqdqp.root.comops.site
build id 4581871
I20260812 05:20:25.842962 2980 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 05:20:25.843163 2980 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 05:20:25.847774 2991 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 05:20:25.847782 2989 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 05:20:25.847831 2980 server_base.cc:1034] running on GCE node
W20260812 05:20:25.847815 2986 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 05:20:25.848084 2980 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 05:20:25.848282 2980 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 05:20:25.849426 2980 hybrid_clock.cc:648] HybridClock initialized: now 1786512025849401 us; error 33 us; skew 500 ppm
Aug 12 05:20:25 dist-test-slave-42z9 krb5kdc[2531](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512025, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.100.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 05:20:25.851807 2980 init.cc:377] Logged in from keytab as kudu/127.31.100.2@KRBTEST.COM (short username kudu)
I20260812 05:20:25.852216 2980 webserver.cc:460] Webserver started at http://127.31.100.2:42223/ using document root <none> and password file <none>
I20260812 05:20:25.852428 2980 fs_manager.cc:359] Metadata directory not provided
I20260812 05:20:25.852478 2980 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 05:20:25.852599 2980 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 05:20:25.853577 2980 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786512017613366-32144-0/minicluster-data/ts-1/data/instance:
uuid: "5b0b9ad99299469c8850bf5f32f62180"
format_stamp: "Formatted at 2026-08-12 05:20:25 on dist-test-slave-42z9"
server_key: "da97b91d3f43bf721085c46c46545e81"
server_key_iv: "bf5337a3ab3b858c5ac45a6b3cef77df"
server_key_version: "encryptionkey@0"
I20260812 05:20:25.853885 2980 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786512017613366-32144-0/minicluster-data/ts-1/wal/instance:
uuid: "5b0b9ad99299469c8850bf5f32f62180"
format_stamp: "Formatted at 2026-08-12 05:20:25 on dist-test-slave-42z9"
server_key: "da97b91d3f43bf721085c46c46545e81"
server_key_iv: "bf5337a3ab3b858c5ac45a6b3cef77df"
server_key_version: "encryptionkey@0"
I20260812 05:20:25.855183 2980 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260812 05:20:25.855813 2999 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 05:20:25.855953 2980 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260812 05:20:25.856042 2980 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786512017613366-32144-0/minicluster-data/ts-1/data,/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786512017613366-32144-0/minicluster-data/ts-1/wal
uuid: "5b0b9ad99299469c8850bf5f32f62180"
format_stamp: "Formatted at 2026-08-12 05:20:25 on dist-test-slave-42z9"
server_key: "da97b91d3f43bf721085c46c46545e81"
server_key_iv: "bf5337a3ab3b858c5ac45a6b3cef77df"
server_key_version: "encryptionkey@0"
I20260812 05:20:25.856101 2980 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786512017613366-32144-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786512017613366-32144-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786512017613366-32144-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 05:20:25.867293 2980 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 05:20:25.868837 2980 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 05:20:25.868968 2980 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 05:20:25.869424 2980 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 05:20:25.869493 2980 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 05:20:25.869519 2980 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 05:20:25.869547 2980 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 12 05:20:25 dist-test-slave-42z9 krb5kdc[2531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512025, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.100.2@KRBTEST.COM for kudu/127.31.100.62@KRBTEST.COM
I20260812 05:20:25.876354 2980 rpc_server.cc:307] RPC server started. Bound to: 127.31.100.2:33141
I20260812 05:20:25.876447 3189 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.100.2:33141 every 8 connection(s)
I20260812 05:20:25.876726 2980 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786512017613366-32144-0/minicluster-data/ts-1/data/info.pb
I20260812 05:20:25.878407 2688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:25.869800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.100.2:37887 (local address 127.31.100.62:45993)
0812 05:20:25.869847 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0812 05:20:25.869849 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 05:20:25.870057 (+ 208us) server_negotiation.cc:407] Connection header received
0812 05:20:25.870512 (+ 455us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:20:25.870514 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:20:25.870527 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:20:25.870596 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:20:25.871106 (+ 510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:25.871864 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:25.872871 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:25.873165 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:25.873244 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:25.875912 (+ 2668us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 05:20:25.875914 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 05:20:25.875914 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 05:20:25.875922 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0812 05:20:25.877348 (+ 1426us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:20:25.877721 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:20:25.877722 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:20:25.877722 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:20:25.877771 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:20:25.877940 (+ 169us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:20:25.877941 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:20:25.877941 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:20:25.878032 (+ 91us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 05:20:25.878076 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0812 05:20:25.878261 (+ 185us) server_negotiation.cc:299] Negotiation successful
0812 05:20:25.878315 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260812 05:20:25.878515 3017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:25.869799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.62:45993 (local address 127.31.100.2:37887)
0812 05:20:25.869944 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0812 05:20:25.869977 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0812 05:20:25.870415 (+ 438us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:20:25.870643 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:20:25.870646 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:20:25.870860 (+ 214us) client_negotiation.cc:189] Negotiated authn=SASL
0812 05:20:25.871025 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:20:25.871029 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:20:25.871959 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:20:25.871960 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:20:25.872801 (+ 841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:20:25.872803 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:20:25.873252 (+ 449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:20:25.873253 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:20:25.873375 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:25.873858 (+ 483us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 05:20:25.873867 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0812 05:20:25.875840 (+ 1973us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 05:20:25.877420 (+ 1580us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 05:20:25.877423 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 05:20:25.877425 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 05:20:25.877660 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 05:20:25.877813 (+ 153us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 05:20:25.877814 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 05:20:25.877814 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 05:20:25.877900 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 05:20:25.878075 (+ 175us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 05:20:25.878078 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 05:20:25.878176 (+ 98us) client_negotiation.cc:769] Sending connection context
0812 05:20:25.878249 (+ 73us) client_negotiation.cc:240] Negotiation successful
0812 05:20:25.878356 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":75,"threads_started":1}
I20260812 05:20:25.878943 3191 heartbeater.cc:347] Connected to a master server at 127.31.100.62:45993
I20260812 05:20:25.879024 3191 heartbeater.cc:464] Registering TS with master...
I20260812 05:20:25.879199 3191 heartbeater.cc:511] Master 127.31.100.62:45993 requested a full tablet report, sending...
I20260812 05:20:25.879550 2607 ts_manager.cc:194] Registered new tserver with Master: 5b0b9ad99299469c8850bf5f32f62180 (127.31.100.2:33141)
I20260812 05:20:25.879930 2607 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.100.2@KRBTEST.COM'} at 127.31.100.2:37887
I20260812 05:20:25.884222 3202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:25.879882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.62:45993 (local address 127.31.100.2:46917)
0812 05:20:25.879999 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0812 05:20:25.880008 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0812 05:20:25.880064 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:20:25.880167 (+ 103us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:20:25.880167 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:20:25.880342 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0812 05:20:25.880392 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:20:25.880393 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:20:25.881045 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:20:25.881045 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:20:25.881602 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:20:25.881604 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:20:25.881867 (+ 263us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:20:25.881867 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:20:25.881915 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:25.882205 (+ 290us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 05:20:25.882208 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0812 05:20:25.882439 (+ 231us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 05:20:25.883514 (+ 1075us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 05:20:25.883515 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 05:20:25.883515 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 05:20:25.883666 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 05:20:25.883793 (+ 127us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 05:20:25.883793 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 05:20:25.883793 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 05:20:25.883833 (+ 40us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 05:20:25.884023 (+ 190us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 05:20:25.884024 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 05:20:25.884056 (+ 32us) client_negotiation.cc:769] Sending connection context
0812 05:20:25.884098 (+ 42us) client_negotiation.cc:240] Negotiation successful
0812 05:20:25.884164 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":72,"threads_started":1}
I20260812 05:20:25.884233 2688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:25.879949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.100.2:46917 (local address 127.31.100.62:45993)
0812 05:20:25.879986 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0812 05:20:25.879987 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0812 05:20:25.880055 (+ 68us) server_negotiation.cc:407] Connection header received
0812 05:20:25.880107 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:20:25.880107 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:20:25.880116 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:20:25.880162 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:20:25.880463 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:25.880975 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:25.881655 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:25.881820 (+ 165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:25.881863 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:25.882500 (+ 637us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 05:20:25.882501 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 05:20:25.882501 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 05:20:25.882506 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0812 05:20:25.883456 (+ 950us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:20:25.883717 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:20:25.883718 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:20:25.883718 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:20:25.883759 (+ 41us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:20:25.883864 (+ 105us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:20:25.883864 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:20:25.883864 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:20:25.883982 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 05:20:25.884026 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0812 05:20:25.884122 (+ 96us) server_negotiation.cc:299] Negotiation successful
0812 05:20:25.884180 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260812 05:20:25.884549 32144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0waFvT/build/release/bin/kudu as pid 2980
I20260812 05:20:25.884631 32144 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786512017613366-32144-0/minicluster-data/ts-1/wal/instance
I20260812 05:20:25.884819 32144 external_mini_cluster.cc:1428] Setting key f0bd9337156995583aafee466c7e74ab
WARNING: no policy specified for kudu/127.31.100.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.100.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.100.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.3.keytab.
Entry for principal kudu/127.31.100.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.3.keytab.
I20260812 05:20:25.906123 32144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0waFvT/build/release/bin/kudu
/tmp/dist-test-task0waFvT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786512017613366-32144-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786512017613366-32144-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786512017613366-32144-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786512017613366-32144-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.100.3:0
--local_ip_for_outbound_sockets=127.31.100.3
--webserver_interface=127.31.100.3
--webserver_port=0
--tserver_master_addrs=127.31.100.62:45993
--keytab_file=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.3.keytab
--principal=kudu/127.31.100.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.100.20:34405
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786512017613366-32144-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786512017613366-32144-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786512017613366-32144-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 05:20:25.985255 3208 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 05:20:25.985460 3208 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 05:20:25.985495 3208 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 05:20:25.986585 3208 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 05:20:25.986620 3208 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 05:20:25.986639 3208 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260812 05:20:25.986649 3208 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260812 05:20:25.986665 3208 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.100.3
I20260812 05:20:25.988029 3208 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.100.20:34405
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786512017613366-32144-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786512017613366-32144-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.3.keytab
--principal=kudu/127.31.100.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.100.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786512017613366-32144-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.100.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.31.100.62:45993
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3208
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.100.3
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786512017613366-32144-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 04:20:34 UTC on ccycloud.quasar-phqdqp.root.comops.site
build id 4581871
I20260812 05:20:25.988198 3208 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 05:20:25.988365 3208 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 05:20:25.992966 3219 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 05:20:25.992972 3216 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 05:20:25.992967 3215 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 05:20:25.993175 3208 server_base.cc:1034] running on GCE node
I20260812 05:20:25.993358 3208 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 05:20:25.993619 3208 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 05:20:25.994761 3208 hybrid_clock.cc:648] HybridClock initialized: now 1786512025994742 us; error 35 us; skew 500 ppm
Aug 12 05:20:25 dist-test-slave-42z9 krb5kdc[2531](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512025, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.100.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 05:20:25.997114 3208 init.cc:377] Logged in from keytab as kudu/127.31.100.3@KRBTEST.COM (short username kudu)
I20260812 05:20:25.997563 3208 webserver.cc:460] Webserver started at http://127.31.100.3:43127/ using document root <none> and password file <none>
I20260812 05:20:25.997756 3208 fs_manager.cc:359] Metadata directory not provided
I20260812 05:20:25.997803 3208 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 05:20:25.997931 3208 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 05:20:25.998992 3208 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786512017613366-32144-0/minicluster-data/ts-2/data/instance:
uuid: "04f98f06eae54d60b1f5bdb390d4b9ef"
format_stamp: "Formatted at 2026-08-12 05:20:25 on dist-test-slave-42z9"
server_key: "7c9fc7a94e82804758b3f3fed1a2b809"
server_key_iv: "979f4178a7af74f393dc662326e60566"
server_key_version: "encryptionkey@0"
I20260812 05:20:25.999312 3208 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786512017613366-32144-0/minicluster-data/ts-2/wal/instance:
uuid: "04f98f06eae54d60b1f5bdb390d4b9ef"
format_stamp: "Formatted at 2026-08-12 05:20:25 on dist-test-slave-42z9"
server_key: "7c9fc7a94e82804758b3f3fed1a2b809"
server_key_iv: "979f4178a7af74f393dc662326e60566"
server_key_version: "encryptionkey@0"
I20260812 05:20:26.000572 3208 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260812 05:20:26.001205 3231 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 05:20:26.001362 3208 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260812 05:20:26.001466 3208 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786512017613366-32144-0/minicluster-data/ts-2/data,/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786512017613366-32144-0/minicluster-data/ts-2/wal
uuid: "04f98f06eae54d60b1f5bdb390d4b9ef"
format_stamp: "Formatted at 2026-08-12 05:20:25 on dist-test-slave-42z9"
server_key: "7c9fc7a94e82804758b3f3fed1a2b809"
server_key_iv: "979f4178a7af74f393dc662326e60566"
server_key_version: "encryptionkey@0"
I20260812 05:20:26.001525 3208 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786512017613366-32144-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786512017613366-32144-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786512017613366-32144-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 05:20:26.011019 3208 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 05:20:26.012975 3208 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 05:20:26.013118 3208 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 05:20:26.013682 3208 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 05:20:26.013723 3208 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 05:20:26.013743 3208 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 05:20:26.013754 3208 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 12 05:20:26 dist-test-slave-42z9 krb5kdc[2531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512025, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.100.3@KRBTEST.COM for kudu/127.31.100.62@KRBTEST.COM
I20260812 05:20:26.019848 3208 rpc_server.cc:307] RPC server started. Bound to: 127.31.100.3:37167
I20260812 05:20:26.020026 3422 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.100.3:37167 every 8 connection(s)
I20260812 05:20:26.020247 3208 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786512017613366-32144-0/minicluster-data/ts-2/data/info.pb
I20260812 05:20:26.022940 2688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:26.014021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.100.3:58473 (local address 127.31.100.62:45993)
0812 05:20:26.014073 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0812 05:20:26.014076 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 05:20:26.014294 (+ 218us) server_negotiation.cc:407] Connection header received
0812 05:20:26.014737 (+ 443us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:20:26.014738 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:20:26.014752 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:20:26.014832 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:20:26.015391 (+ 559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:26.016154 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:26.017636 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:26.017927 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:26.017990 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:26.020309 (+ 2319us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 05:20:26.020310 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 05:20:26.020311 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 05:20:26.020318 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 05:20:26.021717 (+ 1399us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:20:26.022069 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:20:26.022070 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:20:26.022070 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:20:26.022128 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:20:26.022337 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:20:26.022339 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:20:26.022339 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:20:26.022449 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 05:20:26.022503 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0812 05:20:26.022767 (+ 264us) server_negotiation.cc:299] Negotiation successful
0812 05:20:26.022852 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260812 05:20:26.023090 3244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:26.014007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.62:45993 (local address 127.31.100.3:58473)
0812 05:20:26.014175 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0812 05:20:26.014213 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0812 05:20:26.014651 (+ 438us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:20:26.014882 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:20:26.014886 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:20:26.015126 (+ 240us) client_negotiation.cc:189] Negotiated authn=SASL
0812 05:20:26.015317 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:20:26.015321 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:20:26.016250 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:20:26.016251 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:20:26.017538 (+ 1287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:20:26.017541 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:20:26.018003 (+ 462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:20:26.018004 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:20:26.018137 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:26.018597 (+ 460us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 05:20:26.018607 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0812 05:20:26.020220 (+ 1613us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 05:20:26.021788 (+ 1568us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 05:20:26.021794 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 05:20:26.021796 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 05:20:26.021997 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 05:20:26.022181 (+ 184us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 05:20:26.022181 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 05:20:26.022181 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 05:20:26.022283 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 05:20:26.022503 (+ 220us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 05:20:26.022509 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 05:20:26.022635 (+ 126us) client_negotiation.cc:769] Sending connection context
0812 05:20:26.022734 (+ 99us) client_negotiation.cc:240] Negotiation successful
0812 05:20:26.022881 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":70,"threads_started":1}
I20260812 05:20:26.023509 3423 heartbeater.cc:347] Connected to a master server at 127.31.100.62:45993
I20260812 05:20:26.023594 3423 heartbeater.cc:464] Registering TS with master...
I20260812 05:20:26.023768 3423 heartbeater.cc:511] Master 127.31.100.62:45993 requested a full tablet report, sending...
I20260812 05:20:26.024129 2606 ts_manager.cc:194] Registered new tserver with Master: 04f98f06eae54d60b1f5bdb390d4b9ef (127.31.100.3:37167)
I20260812 05:20:26.024564 2606 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.100.3@KRBTEST.COM'} at 127.31.100.3:58473
I20260812 05:20:26.029137 2688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:26.024556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.100.3:58191 (local address 127.31.100.62:45993)
0812 05:20:26.024597 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0812 05:20:26.024598 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0812 05:20:26.024705 (+ 107us) server_negotiation.cc:407] Connection header received
0812 05:20:26.024763 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:20:26.024763 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:20:26.024773 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:20:26.024814 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:20:26.025107 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:26.025684 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:26.026297 (+ 613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:26.026479 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:26.026539 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:26.027249 (+ 710us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 05:20:26.027250 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 05:20:26.027250 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 05:20:26.027255 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0812 05:20:26.028392 (+ 1137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:20:26.028655 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:20:26.028655 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:20:26.028656 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:20:26.028699 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:20:26.028824 (+ 125us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:20:26.028824 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:20:26.028824 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:20:26.028900 (+ 76us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 05:20:26.028939 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0812 05:20:26.029032 (+ 93us) server_negotiation.cc:299] Negotiation successful
0812 05:20:26.029083 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260812 05:20:26.029150 3430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:26.024488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.62:45993 (local address 127.31.100.3:58191)
0812 05:20:26.024645 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0812 05:20:26.024655 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0812 05:20:26.024718 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:20:26.024818 (+ 100us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:20:26.024819 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:20:26.024986 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0812 05:20:26.025041 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:20:26.025043 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:20:26.025764 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:20:26.025765 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:20:26.026245 (+ 480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:20:26.026246 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:20:26.026543 (+ 297us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:20:26.026543 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:20:26.026601 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:26.026936 (+ 335us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 05:20:26.026940 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0812 05:20:26.027185 (+ 245us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 05:20:26.028455 (+ 1270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 05:20:26.028456 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 05:20:26.028456 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 05:20:26.028603 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 05:20:26.028743 (+ 140us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 05:20:26.028744 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 05:20:26.028744 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 05:20:26.028790 (+ 46us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 05:20:26.028938 (+ 148us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 05:20:26.028938 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 05:20:26.028971 (+ 33us) client_negotiation.cc:769] Sending connection context
0812 05:20:26.029013 (+ 42us) client_negotiation.cc:240] Negotiation successful
0812 05:20:26.029083 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":102,"threads_started":1}
I20260812 05:20:26.030373 32144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0waFvT/build/release/bin/kudu as pid 3208
I20260812 05:20:26.030474 32144 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786512017613366-32144-0/minicluster-data/ts-2/wal/instance
I20260812 05:20:26.030648 32144 external_mini_cluster.cc:1428] Setting key 56b5ed8364a8aa6d7299d9d4fb889223
I20260812 05:20:26.031400 32144 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 12 05:20:26 dist-test-slave-42z9 krb5kdc[2531](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512026, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 12 05:20:26 dist-test-slave-42z9 krb5kdc[2531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512026, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.100.62@KRBTEST.COM
I20260812 05:20:26.054039 2688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:26.047420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44252 (local address 127.31.100.62:45993)
0812 05:20:26.047471 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0812 05:20:26.047473 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 05:20:26.047482 (+ 9us) server_negotiation.cc:407] Connection header received
0812 05:20:26.047504 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:20:26.047505 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:20:26.047513 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:20:26.047559 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:20:26.048111 (+ 552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:26.048607 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:26.049321 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:26.049556 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:26.049606 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:26.051720 (+ 2114us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 05:20:26.051722 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 05:20:26.051722 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 05:20:26.051728 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0812 05:20:26.052947 (+ 1219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:20:26.053274 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:20:26.053275 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:20:26.053276 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:20:26.053330 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:20:26.053531 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:20:26.053532 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:20:26.053533 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:20:26.053641 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 05:20:26.053725 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0812 05:20:26.053837 (+ 112us) server_negotiation.cc:299] Negotiation successful
0812 05:20:26.053910 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260812 05:20:26.055533 2606 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44252:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 05:20:26.057050 2606 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 05:20:26.063287 3445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:26.059136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.2:33141 (local address 127.0.0.1:54216)
0812 05:20:26.059376 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0812 05:20:26.059414 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0812 05:20:26.059482 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:20:26.059904 (+ 422us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:20:26.059905 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:20:26.059907 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 05:20:26.059945 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:20:26.059946 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:20:26.061229 (+ 1283us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:20:26.061230 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:20:26.062330 (+ 1100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:20:26.062332 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:20:26.062992 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:20:26.062993 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:20:26.063073 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:26.063086 (+ 13us) client_negotiation.cc:769] Sending connection context
0812 05:20:26.063153 (+ 67us) client_negotiation.cc:240] Negotiation successful
0812 05:20:26.063208 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":52,"threads_started":1}
I20260812 05:20:26.063302 3444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:26.059233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54216 (local address 127.31.100.2:33141)
0812 05:20:26.059619 (+ 386us) server_negotiation.cc:206] Beginning negotiation
0812 05:20:26.059624 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 05:20:26.059634 (+ 10us) server_negotiation.cc:407] Connection header received
0812 05:20:26.059692 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:20:26.059695 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:20:26.059711 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:20:26.059791 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 05:20:26.060025 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:26.061145 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:26.062415 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:26.062914 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:26.062989 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:26.063053 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0812 05:20:26.063175 (+ 122us) server_negotiation.cc:299] Negotiation successful
0812 05:20:26.063213 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":347,"thread_start_us":72,"threads_started":1}
I20260812 05:20:26.063555 3442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:26.059145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.1:46417 (local address 127.0.0.1:41550)
0812 05:20:26.059466 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0812 05:20:26.059477 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0812 05:20:26.059529 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:20:26.059839 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:20:26.059842 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:20:26.059844 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 05:20:26.059910 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:20:26.059912 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:20:26.061510 (+ 1598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:20:26.061511 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:20:26.062347 (+ 836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:20:26.062350 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:20:26.063295 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:20:26.063296 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:20:26.063377 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:26.063382 (+ 5us) client_negotiation.cc:769] Sending connection context
0812 05:20:26.063429 (+ 47us) client_negotiation.cc:240] Negotiation successful
0812 05:20:26.063477 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":305,"mutex_wait_us":1,"thread_start_us":167,"threads_started":1}
I20260812 05:20:26.063807 3448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:26.059484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41550 (local address 127.31.100.1:46417)
0812 05:20:26.059660 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0812 05:20:26.059663 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 05:20:26.059690 (+ 27us) server_negotiation.cc:407] Connection header received
0812 05:20:26.059724 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:20:26.059726 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:20:26.059741 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:20:26.059812 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 05:20:26.059997 (+ 185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:26.061157 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:26.062433 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:26.063137 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:26.063205 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:26.063259 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0812 05:20:26.063668 (+ 409us) server_negotiation.cc:299] Negotiation successful
0812 05:20:26.063713 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":73,"threads_started":1}
I20260812 05:20:26.063815 3446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:26.059419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39414 (local address 127.31.100.3:37167)
0812 05:20:26.059922 (+ 503us) server_negotiation.cc:206] Beginning negotiation
0812 05:20:26.059926 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 05:20:26.059936 (+ 10us) server_negotiation.cc:407] Connection header received
0812 05:20:26.059970 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:20:26.059972 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:20:26.059986 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:20:26.060053 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 05:20:26.060227 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:26.060892 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:26.062407 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:26.063144 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:26.063205 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:26.063259 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0812 05:20:26.063692 (+ 433us) server_negotiation.cc:299] Negotiation successful
0812 05:20:26.063731 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":466,"thread_start_us":80,"threads_started":1}
I20260812 05:20:26.064009 3443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:26.059147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.3:37167 (local address 127.0.0.1:39414)
0812 05:20:26.059580 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0812 05:20:26.059589 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0812 05:20:26.059626 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:20:26.060117 (+ 491us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:20:26.060117 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:20:26.060120 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 05:20:26.060162 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:20:26.060163 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:20:26.060967 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:20:26.060967 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:20:26.062309 (+ 1342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:20:26.062313 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:20:26.063295 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:20:26.063296 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:20:26.063542 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:26.063546 (+ 4us) client_negotiation.cc:769] Sending connection context
0812 05:20:26.063591 (+ 45us) client_negotiation.cc:240] Negotiation successful
0812 05:20:26.063947 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":423,"mutex_wait_us":71,"thread_start_us":63,"threads_started":1}
I20260812 05:20:26.064038 3086 tablet_service.cc:1467] Processing CreateTablet for tablet 37bc9f2b7a4f4724a0d47e6e11ea2053 (DEFAULT_TABLE table=test-table [id=4efdb4056a394e4dbbe277f6abbf106b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 05:20:26.064459 3086 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 37bc9f2b7a4f4724a0d47e6e11ea2053. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 05:20:26.064424 2850 tablet_service.cc:1467] Processing CreateTablet for tablet 37bc9f2b7a4f4724a0d47e6e11ea2053 (DEFAULT_TABLE table=test-table [id=4efdb4056a394e4dbbe277f6abbf106b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 05:20:26.064774 2850 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 37bc9f2b7a4f4724a0d47e6e11ea2053. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 05:20:26.064736 3303 tablet_service.cc:1467] Processing CreateTablet for tablet 37bc9f2b7a4f4724a0d47e6e11ea2053 (DEFAULT_TABLE table=test-table [id=4efdb4056a394e4dbbe277f6abbf106b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 05:20:26.065066 3303 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 37bc9f2b7a4f4724a0d47e6e11ea2053. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 05:20:26.067569 3449 tablet_bootstrap.cc:492] T 37bc9f2b7a4f4724a0d47e6e11ea2053 P 5b0b9ad99299469c8850bf5f32f62180: Bootstrap starting.
I20260812 05:20:26.068061 3450 tablet_bootstrap.cc:492] T 37bc9f2b7a4f4724a0d47e6e11ea2053 P 04f98f06eae54d60b1f5bdb390d4b9ef: Bootstrap starting.
I20260812 05:20:26.068410 3449 tablet_bootstrap.cc:654] T 37bc9f2b7a4f4724a0d47e6e11ea2053 P 5b0b9ad99299469c8850bf5f32f62180: Neither blocks nor log segments found. Creating new log.
I20260812 05:20:26.068434 3452 tablet_bootstrap.cc:492] T 37bc9f2b7a4f4724a0d47e6e11ea2053 P 4dce95e2db6d454f9a636ef4f25cd23b: Bootstrap starting.
I20260812 05:20:26.068874 3449 log.cc:824] T 37bc9f2b7a4f4724a0d47e6e11ea2053 P 5b0b9ad99299469c8850bf5f32f62180: Log is configured to *not* fsync() on all Append() calls
I20260812 05:20:26.069011 3450 tablet_bootstrap.cc:654] T 37bc9f2b7a4f4724a0d47e6e11ea2053 P 04f98f06eae54d60b1f5bdb390d4b9ef: Neither blocks nor log segments found. Creating new log.
I20260812 05:20:26.069106 3452 tablet_bootstrap.cc:654] T 37bc9f2b7a4f4724a0d47e6e11ea2053 P 4dce95e2db6d454f9a636ef4f25cd23b: Neither blocks nor log segments found. Creating new log.
I20260812 05:20:26.069324 3450 log.cc:824] T 37bc9f2b7a4f4724a0d47e6e11ea2053 P 04f98f06eae54d60b1f5bdb390d4b9ef: Log is configured to *not* fsync() on all Append() calls
I20260812 05:20:26.069571 3452 log.cc:824] T 37bc9f2b7a4f4724a0d47e6e11ea2053 P 4dce95e2db6d454f9a636ef4f25cd23b: Log is configured to *not* fsync() on all Append() calls
I20260812 05:20:26.069814 3449 tablet_bootstrap.cc:492] T 37bc9f2b7a4f4724a0d47e6e11ea2053 P 5b0b9ad99299469c8850bf5f32f62180: No bootstrap required, opened a new log
I20260812 05:20:26.069902 3449 ts_tablet_manager.cc:1397] T 37bc9f2b7a4f4724a0d47e6e11ea2053 P 5b0b9ad99299469c8850bf5f32f62180: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260812 05:20:26.070073 3450 tablet_bootstrap.cc:492] T 37bc9f2b7a4f4724a0d47e6e11ea2053 P 04f98f06eae54d60b1f5bdb390d4b9ef: No bootstrap required, opened a new log
I20260812 05:20:26.070133 3450 ts_tablet_manager.cc:1397] T 37bc9f2b7a4f4724a0d47e6e11ea2053 P 04f98f06eae54d60b1f5bdb390d4b9ef: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260812 05:20:26.070263 3452 tablet_bootstrap.cc:492] T 37bc9f2b7a4f4724a0d47e6e11ea2053 P 4dce95e2db6d454f9a636ef4f25cd23b: No bootstrap required, opened a new log
I20260812 05:20:26.070333 3452 ts_tablet_manager.cc:1397] T 37bc9f2b7a4f4724a0d47e6e11ea2053 P 4dce95e2db6d454f9a636ef4f25cd23b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260812 05:20:26.071663 3450 raft_consensus.cc:348] T 37bc9f2b7a4f4724a0d47e6e11ea2053 P 04f98f06eae54d60b1f5bdb390d4b9ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b0b9ad99299469c8850bf5f32f62180" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 33141 } } peers { permanent_uuid: "04f98f06eae54d60b1f5bdb390d4b9ef" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 37167 } } peers { permanent_uuid: "4dce95e2db6d454f9a636ef4f25cd23b" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 46417 } }
I20260812 05:20:26.071791 3450 raft_consensus.cc:374] T 37bc9f2b7a4f4724a0d47e6e11ea2053 P 04f98f06eae54d60b1f5bdb390d4b9ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 05:20:26.071823 3450 raft_consensus.cc:729] T 37bc9f2b7a4f4724a0d47e6e11ea2053 P 04f98f06eae54d60b1f5bdb390d4b9ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04f98f06eae54d60b1f5bdb390d4b9ef, State: Initialized, Role: FOLLOWER
I20260812 05:20:26.071774 3449 raft_consensus.cc:348] T 37bc9f2b7a4f4724a0d47e6e11ea2053 P 5b0b9ad99299469c8850bf5f32f62180 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b0b9ad99299469c8850bf5f32f62180" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 33141 } } peers { permanent_uuid: "04f98f06eae54d60b1f5bdb390d4b9ef" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 37167 } } peers { permanent_uuid: "4dce95e2db6d454f9a636ef4f25cd23b" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 46417 } }
I20260812 05:20:26.071934 3449 raft_consensus.cc:374] T 37bc9f2b7a4f4724a0d47e6e11ea2053 P 5b0b9ad99299469c8850bf5f32f62180 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 05:20:26.071981 3449 raft_consensus.cc:729] T 37bc9f2b7a4f4724a0d47e6e11ea2053 P 5b0b9ad99299469c8850bf5f32f62180 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b0b9ad99299469c8850bf5f32f62180, State: Initialized, Role: FOLLOWER
I20260812 05:20:26.071964 3450 consensus_queue.cc:260] T 37bc9f2b7a4f4724a0d47e6e11ea2053 P 04f98f06eae54d60b1f5bdb390d4b9ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b0b9ad99299469c8850bf5f32f62180" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 33141 } } peers { permanent_uuid: "04f98f06eae54d60b1f5bdb390d4b9ef" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 37167 } } peers { permanent_uuid: "4dce95e2db6d454f9a636ef4f25cd23b" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 46417 } }
I20260812 05:20:26.072108 3449 consensus_queue.cc:260] T 37bc9f2b7a4f4724a0d47e6e11ea2053 P 5b0b9ad99299469c8850bf5f32f62180 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b0b9ad99299469c8850bf5f32f62180" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 33141 } } peers { permanent_uuid: "04f98f06eae54d60b1f5bdb390d4b9ef" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 37167 } } peers { permanent_uuid: "4dce95e2db6d454f9a636ef4f25cd23b" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 46417 } }
I20260812 05:20:26.072149 3452 raft_consensus.cc:348] T 37bc9f2b7a4f4724a0d47e6e11ea2053 P 4dce95e2db6d454f9a636ef4f25cd23b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b0b9ad99299469c8850bf5f32f62180" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 33141 } } peers { permanent_uuid: "04f98f06eae54d60b1f5bdb390d4b9ef" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 37167 } } peers { permanent_uuid: "4dce95e2db6d454f9a636ef4f25cd23b" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 46417 } }
I20260812 05:20:26.072235 3450 ts_tablet_manager.cc:1428] T 37bc9f2b7a4f4724a0d47e6e11ea2053 P 04f98f06eae54d60b1f5bdb390d4b9ef: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260812 05:20:26.072300 3452 raft_consensus.cc:374] T 37bc9f2b7a4f4724a0d47e6e11ea2053 P 4dce95e2db6d454f9a636ef4f25cd23b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 05:20:26.072299 3423 heartbeater.cc:503] Master 127.31.100.62:45993 was elected leader, sending a full tablet report...
I20260812 05:20:26.072337 3452 raft_consensus.cc:729] T 37bc9f2b7a4f4724a0d47e6e11ea2053 P 4dce95e2db6d454f9a636ef4f25cd23b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4dce95e2db6d454f9a636ef4f25cd23b, State: Initialized, Role: FOLLOWER
I20260812 05:20:26.072377 3449 ts_tablet_manager.cc:1428] T 37bc9f2b7a4f4724a0d47e6e11ea2053 P 5b0b9ad99299469c8850bf5f32f62180: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260812 05:20:26.072453 3452 consensus_queue.cc:260] T 37bc9f2b7a4f4724a0d47e6e11ea2053 P 4dce95e2db6d454f9a636ef4f25cd23b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b0b9ad99299469c8850bf5f32f62180" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 33141 } } peers { permanent_uuid: "04f98f06eae54d60b1f5bdb390d4b9ef" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 37167 } } peers { permanent_uuid: "4dce95e2db6d454f9a636ef4f25cd23b" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 46417 } }
I20260812 05:20:26.072494 3191 heartbeater.cc:503] Master 127.31.100.62:45993 was elected leader, sending a full tablet report...
I20260812 05:20:26.072714 3452 ts_tablet_manager.cc:1428] T 37bc9f2b7a4f4724a0d47e6e11ea2053 P 4dce95e2db6d454f9a636ef4f25cd23b: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260812 05:20:26.072795 2959 heartbeater.cc:503] Master 127.31.100.62:45993 was elected leader, sending a full tablet report...
W20260812 05:20:26.127288 3193 tablet.cc:2369] T 37bc9f2b7a4f4724a0d47e6e11ea2053 P 5b0b9ad99299469c8850bf5f32f62180: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 05:20:26.230068 2961 tablet.cc:2369] T 37bc9f2b7a4f4724a0d47e6e11ea2053 P 4dce95e2db6d454f9a636ef4f25cd23b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 05:20:26.271036 3424 tablet.cc:2369] T 37bc9f2b7a4f4724a0d47e6e11ea2053 P 04f98f06eae54d60b1f5bdb390d4b9ef: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 05:20:26.325778 3456 raft_consensus.cc:482] T 37bc9f2b7a4f4724a0d47e6e11ea2053 P 04f98f06eae54d60b1f5bdb390d4b9ef [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 05:20:26.325907 3456 raft_consensus.cc:504] T 37bc9f2b7a4f4724a0d47e6e11ea2053 P 04f98f06eae54d60b1f5bdb390d4b9ef [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b0b9ad99299469c8850bf5f32f62180" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 33141 } } peers { permanent_uuid: "04f98f06eae54d60b1f5bdb390d4b9ef" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 37167 } } peers { permanent_uuid: "4dce95e2db6d454f9a636ef4f25cd23b" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 46417 } }
I20260812 05:20:26.326282 3456 leader_election.cc:288] T 37bc9f2b7a4f4724a0d47e6e11ea2053 P 04f98f06eae54d60b1f5bdb390d4b9ef [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5b0b9ad99299469c8850bf5f32f62180 (127.31.100.2:33141), 4dce95e2db6d454f9a636ef4f25cd23b (127.31.100.1:46417)
I20260812 05:20:26.329238 3244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:26.326485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.2:33141 (local address 127.31.100.3:52483)
0812 05:20:26.326534 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0812 05:20:26.326545 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0812 05:20:26.326591 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:20:26.326748 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:20:26.326749 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:20:26.326751 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 05:20:26.326817 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:20:26.326819 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:20:26.327562 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:20:26.327563 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:20:26.328463 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:20:26.328466 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:20:26.329010 (+ 544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:20:26.329011 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:20:26.329082 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:26.329085 (+ 3us) client_negotiation.cc:769] Sending connection context
0812 05:20:26.329134 (+ 49us) client_negotiation.cc:240] Negotiation successful
0812 05:20:26.329168 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28,"mutex_wait_us":24}
I20260812 05:20:26.329257 3444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:26.326575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.100.3:52483 (local address 127.31.100.2:33141)
0812 05:20:26.326629 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0812 05:20:26.326630 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0812 05:20:26.326639 (+ 9us) server_negotiation.cc:407] Connection header received
0812 05:20:26.326665 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:20:26.326666 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:20:26.326674 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:20:26.326721 (+ 47us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 05:20:26.326891 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:26.327474 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:26.328550 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:26.328938 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:26.328991 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:26.328998 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0812 05:20:26.329146 (+ 148us) server_negotiation.cc:299] Negotiation successful
0812 05:20:26.329188 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260812 05:20:26.329319 3448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:26.326520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.100.3:59543 (local address 127.31.100.1:46417)
0812 05:20:26.326563 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0812 05:20:26.326565 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 05:20:26.326864 (+ 299us) server_negotiation.cc:407] Connection header received
0812 05:20:26.326921 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:20:26.326922 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:20:26.326932 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:20:26.326973 (+ 41us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 05:20:26.327151 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:26.327660 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:26.328645 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:26.329009 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:26.329075 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:26.329083 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0812 05:20:26.329227 (+ 144us) server_negotiation.cc:299] Negotiation successful
0812 05:20:26.329263 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260812 05:20:26.329352 3460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:26.326485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.1:46417 (local address 127.31.100.3:59543)
0812 05:20:26.326787 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0812 05:20:26.326796 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0812 05:20:26.326852 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:20:26.326980 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:20:26.326981 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:20:26.326983 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 05:20:26.327054 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:20:26.327055 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:20:26.327739 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:20:26.327740 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:20:26.328566 (+ 826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:20:26.328568 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:20:26.329084 (+ 516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:20:26.329085 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:20:26.329158 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:26.329162 (+ 4us) client_negotiation.cc:769] Sending connection context
0812 05:20:26.329214 (+ 52us) client_negotiation.cc:240] Negotiation successful
0812 05:20:26.329252 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":281,"mutex_wait_us":143,"thread_start_us":88,"threads_started":1}
I20260812 05:20:26.329665 3115 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37bc9f2b7a4f4724a0d47e6e11ea2053" candidate_uuid: "04f98f06eae54d60b1f5bdb390d4b9ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b0b9ad99299469c8850bf5f32f62180" is_pre_election: true
I20260812 05:20:26.329715 2878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37bc9f2b7a4f4724a0d47e6e11ea2053" candidate_uuid: "04f98f06eae54d60b1f5bdb390d4b9ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4dce95e2db6d454f9a636ef4f25cd23b" is_pre_election: true
I20260812 05:20:26.329819 3115 raft_consensus.cc:2454] T 37bc9f2b7a4f4724a0d47e6e11ea2053 P 5b0b9ad99299469c8850bf5f32f62180 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 04f98f06eae54d60b1f5bdb390d4b9ef in term 0.
I20260812 05:20:26.329865 2878 raft_consensus.cc:2454] T 37bc9f2b7a4f4724a0d47e6e11ea2053 P 4dce95e2db6d454f9a636ef4f25cd23b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 04f98f06eae54d60b1f5bdb390d4b9ef in term 0.
I20260812 05:20:26.330107 3237 leader_election.cc:302] T 37bc9f2b7a4f4724a0d47e6e11ea2053 P 04f98f06eae54d60b1f5bdb390d4b9ef [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 04f98f06eae54d60b1f5bdb390d4b9ef, 4dce95e2db6d454f9a636ef4f25cd23b; no voters:
I20260812 05:20:26.330271 3456 raft_consensus.cc:2781] T 37bc9f2b7a4f4724a0d47e6e11ea2053 P 04f98f06eae54d60b1f5bdb390d4b9ef [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 05:20:26.330313 3456 raft_consensus.cc:482] T 37bc9f2b7a4f4724a0d47e6e11ea2053 P 04f98f06eae54d60b1f5bdb390d4b9ef [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 05:20:26.330336 3456 raft_consensus.cc:3037] T 37bc9f2b7a4f4724a0d47e6e11ea2053 P 04f98f06eae54d60b1f5bdb390d4b9ef [term 0 FOLLOWER]: Advancing to term 1
I20260812 05:20:26.331305 3456 raft_consensus.cc:504] T 37bc9f2b7a4f4724a0d47e6e11ea2053 P 04f98f06eae54d60b1f5bdb390d4b9ef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b0b9ad99299469c8850bf5f32f62180" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 33141 } } peers { permanent_uuid: "04f98f06eae54d60b1f5bdb390d4b9ef" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 37167 } } peers { permanent_uuid: "4dce95e2db6d454f9a636ef4f25cd23b" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 46417 } }
I20260812 05:20:26.331434 3456 leader_election.cc:288] T 37bc9f2b7a4f4724a0d47e6e11ea2053 P 04f98f06eae54d60b1f5bdb390d4b9ef [CANDIDATE]: Term 1 election: Requested vote from peers 5b0b9ad99299469c8850bf5f32f62180 (127.31.100.2:33141), 4dce95e2db6d454f9a636ef4f25cd23b (127.31.100.1:46417)
I20260812 05:20:26.331593 3115 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37bc9f2b7a4f4724a0d47e6e11ea2053" candidate_uuid: "04f98f06eae54d60b1f5bdb390d4b9ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b0b9ad99299469c8850bf5f32f62180"
I20260812 05:20:26.331650 2878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37bc9f2b7a4f4724a0d47e6e11ea2053" candidate_uuid: "04f98f06eae54d60b1f5bdb390d4b9ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4dce95e2db6d454f9a636ef4f25cd23b"
I20260812 05:20:26.331678 3115 raft_consensus.cc:3037] T 37bc9f2b7a4f4724a0d47e6e11ea2053 P 5b0b9ad99299469c8850bf5f32f62180 [term 0 FOLLOWER]: Advancing to term 1
I20260812 05:20:26.331734 2878 raft_consensus.cc:3037] T 37bc9f2b7a4f4724a0d47e6e11ea2053 P 4dce95e2db6d454f9a636ef4f25cd23b [term 0 FOLLOWER]: Advancing to term 1
I20260812 05:20:26.332413 3115 raft_consensus.cc:2454] T 37bc9f2b7a4f4724a0d47e6e11ea2053 P 5b0b9ad99299469c8850bf5f32f62180 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 04f98f06eae54d60b1f5bdb390d4b9ef in term 1.
I20260812 05:20:26.332496 2878 raft_consensus.cc:2454] T 37bc9f2b7a4f4724a0d47e6e11ea2053 P 4dce95e2db6d454f9a636ef4f25cd23b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 04f98f06eae54d60b1f5bdb390d4b9ef in term 1.
I20260812 05:20:26.332580 3234 leader_election.cc:302] T 37bc9f2b7a4f4724a0d47e6e11ea2053 P 04f98f06eae54d60b1f5bdb390d4b9ef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 04f98f06eae54d60b1f5bdb390d4b9ef, 5b0b9ad99299469c8850bf5f32f62180; no voters:
I20260812 05:20:26.332665 3456 raft_consensus.cc:2781] T 37bc9f2b7a4f4724a0d47e6e11ea2053 P 04f98f06eae54d60b1f5bdb390d4b9ef [term 1 FOLLOWER]: Leader election won for term 1
I20260812 05:20:26.332790 3456 raft_consensus.cc:686] T 37bc9f2b7a4f4724a0d47e6e11ea2053 P 04f98f06eae54d60b1f5bdb390d4b9ef [term 1 LEADER]: Becoming Leader. State: Replica: 04f98f06eae54d60b1f5bdb390d4b9ef, State: Running, Role: LEADER
I20260812 05:20:26.332873 3456 consensus_queue.cc:237] T 37bc9f2b7a4f4724a0d47e6e11ea2053 P 04f98f06eae54d60b1f5bdb390d4b9ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b0b9ad99299469c8850bf5f32f62180" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 33141 } } peers { permanent_uuid: "04f98f06eae54d60b1f5bdb390d4b9ef" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 37167 } } peers { permanent_uuid: "4dce95e2db6d454f9a636ef4f25cd23b" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 46417 } }
I20260812 05:20:26.333719 2606 catalog_manager.cc:5495] T 37bc9f2b7a4f4724a0d47e6e11ea2053 P 04f98f06eae54d60b1f5bdb390d4b9ef reported cstate change: term changed from 0 to 1, leader changed from <none> to 04f98f06eae54d60b1f5bdb390d4b9ef (127.31.100.3). New cstate: current_term: 1 leader_uuid: "04f98f06eae54d60b1f5bdb390d4b9ef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b0b9ad99299469c8850bf5f32f62180" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 33141 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "04f98f06eae54d60b1f5bdb390d4b9ef" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 37167 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4dce95e2db6d454f9a636ef4f25cd23b" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 46417 } health_report { overall_health: UNKNOWN } } }
Aug 12 05:20:26 dist-test-slave-42z9 krb5kdc[2531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512025, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.100.62@KRBTEST.COM for kudu/127.31.100.62@KRBTEST.COM
I20260812 05:20:26.364266 3468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:26.357993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.62:45993 (local address 127.0.0.1:44260)
0812 05:20:26.358155 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0812 05:20:26.358164 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0812 05:20:26.358226 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:20:26.358344 (+ 118us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:20:26.358345 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:20:26.358664 (+ 319us) client_negotiation.cc:189] Negotiated authn=SASL
0812 05:20:26.358713 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:20:26.358716 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:20:26.359400 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:20:26.359401 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:20:26.359962 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:20:26.359964 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:20:26.360261 (+ 297us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:20:26.360262 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:20:26.360310 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:26.360772 (+ 462us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 05:20:26.360777 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0812 05:20:26.362106 (+ 1329us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 05:20:26.363328 (+ 1222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 05:20:26.363335 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 05:20:26.363337 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 05:20:26.363528 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 05:20:26.363693 (+ 165us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 05:20:26.363693 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 05:20:26.363694 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 05:20:26.363743 (+ 49us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 05:20:26.363940 (+ 197us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 05:20:26.363947 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 05:20:26.363991 (+ 44us) client_negotiation.cc:769] Sending connection context
0812 05:20:26.364049 (+ 58us) client_negotiation.cc:240] Negotiation successful
0812 05:20:26.364159 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":102,"threads_started":1}
I20260812 05:20:26.364317 2688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:26.358052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44260 (local address 127.31.100.62:45993)
0812 05:20:26.358101 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0812 05:20:26.358103 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 05:20:26.358218 (+ 115us) server_negotiation.cc:407] Connection header received
0812 05:20:26.358278 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:20:26.358279 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:20:26.358288 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:20:26.358334 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:20:26.358782 (+ 448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:26.359313 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:26.360037 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:26.360209 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:26.360259 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:26.362182 (+ 1923us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 05:20:26.362183 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 05:20:26.362184 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 05:20:26.362190 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0812 05:20:26.363260 (+ 1070us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:20:26.363590 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:20:26.363591 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:20:26.363591 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:20:26.363641 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:20:26.363791 (+ 150us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:20:26.363792 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:20:26.363792 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:20:26.363890 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 05:20:26.363942 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0812 05:20:26.364077 (+ 135us) server_negotiation.cc:299] Negotiation successful
0812 05:20:26.364221 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":81,"server-negotiator.queue_time_us":27}
I20260812 05:20:26.366241 2606 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.31.100.62@KRBTEST.COM'} at 127.0.0.1:44260:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260812 05:20:26.366439 2606 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 05:20:26.367777 2850 tablet_service.cc:1467] Processing CreateTablet for tablet dfc524201e4d4000beadfabd4b72566b (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=8514ba3cd28e4e83a990567a45274711]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260812 05:20:26.367796 3086 tablet_service.cc:1467] Processing CreateTablet for tablet dfc524201e4d4000beadfabd4b72566b (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=8514ba3cd28e4e83a990567a45274711]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260812 05:20:26.367940 3086 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dfc524201e4d4000beadfabd4b72566b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 05:20:26.367945 2850 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dfc524201e4d4000beadfabd4b72566b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 05:20:26.369033 3449 tablet_bootstrap.cc:492] T dfc524201e4d4000beadfabd4b72566b P 5b0b9ad99299469c8850bf5f32f62180: Bootstrap starting.
I20260812 05:20:26.369139 3303 tablet_service.cc:1467] Processing CreateTablet for tablet dfc524201e4d4000beadfabd4b72566b (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=8514ba3cd28e4e83a990567a45274711]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260812 05:20:26.369287 3303 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dfc524201e4d4000beadfabd4b72566b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 05:20:26.369715 3449 tablet_bootstrap.cc:654] T dfc524201e4d4000beadfabd4b72566b P 5b0b9ad99299469c8850bf5f32f62180: Neither blocks nor log segments found. Creating new log.
I20260812 05:20:26.370373 3449 tablet_bootstrap.cc:492] T dfc524201e4d4000beadfabd4b72566b P 5b0b9ad99299469c8850bf5f32f62180: No bootstrap required, opened a new log
I20260812 05:20:26.370428 3449 ts_tablet_manager.cc:1397] T dfc524201e4d4000beadfabd4b72566b P 5b0b9ad99299469c8850bf5f32f62180: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260812 05:20:26.370585 3449 raft_consensus.cc:348] T dfc524201e4d4000beadfabd4b72566b P 5b0b9ad99299469c8850bf5f32f62180 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dce95e2db6d454f9a636ef4f25cd23b" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 46417 } } peers { permanent_uuid: "5b0b9ad99299469c8850bf5f32f62180" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 33141 } } peers { permanent_uuid: "04f98f06eae54d60b1f5bdb390d4b9ef" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 37167 } }
I20260812 05:20:26.370666 3449 raft_consensus.cc:374] T dfc524201e4d4000beadfabd4b72566b P 5b0b9ad99299469c8850bf5f32f62180 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 05:20:26.370684 3449 raft_consensus.cc:729] T dfc524201e4d4000beadfabd4b72566b P 5b0b9ad99299469c8850bf5f32f62180 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b0b9ad99299469c8850bf5f32f62180, State: Initialized, Role: FOLLOWER
I20260812 05:20:26.370733 3449 consensus_queue.cc:260] T dfc524201e4d4000beadfabd4b72566b P 5b0b9ad99299469c8850bf5f32f62180 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dce95e2db6d454f9a636ef4f25cd23b" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 46417 } } peers { permanent_uuid: "5b0b9ad99299469c8850bf5f32f62180" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 33141 } } peers { permanent_uuid: "04f98f06eae54d60b1f5bdb390d4b9ef" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 37167 } }
I20260812 05:20:26.370807 3450 tablet_bootstrap.cc:492] T dfc524201e4d4000beadfabd4b72566b P 04f98f06eae54d60b1f5bdb390d4b9ef: Bootstrap starting.
I20260812 05:20:26.370838 3452 tablet_bootstrap.cc:492] T dfc524201e4d4000beadfabd4b72566b P 4dce95e2db6d454f9a636ef4f25cd23b: Bootstrap starting.
I20260812 05:20:26.370843 3449 ts_tablet_manager.cc:1428] T dfc524201e4d4000beadfabd4b72566b P 5b0b9ad99299469c8850bf5f32f62180: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260812 05:20:26.370990 3458 tablet_replica.cc:439] T dfc524201e4d4000beadfabd4b72566b P 5b0b9ad99299469c8850bf5f32f62180: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dce95e2db6d454f9a636ef4f25cd23b" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 46417 } } peers { permanent_uuid: "5b0b9ad99299469c8850bf5f32f62180" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 33141 } } peers { permanent_uuid: "04f98f06eae54d60b1f5bdb390d4b9ef" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 37167 } } }
I20260812 05:20:26.371078 3458 tablet_replica.cc:442] T dfc524201e4d4000beadfabd4b72566b P 5b0b9ad99299469c8850bf5f32f62180: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 05:20:26.371212 3452 tablet_bootstrap.cc:654] T dfc524201e4d4000beadfabd4b72566b P 4dce95e2db6d454f9a636ef4f25cd23b: Neither blocks nor log segments found. Creating new log.
I20260812 05:20:26.371245 3450 tablet_bootstrap.cc:654] T dfc524201e4d4000beadfabd4b72566b P 04f98f06eae54d60b1f5bdb390d4b9ef: Neither blocks nor log segments found. Creating new log.
I20260812 05:20:26.371800 3452 tablet_bootstrap.cc:492] T dfc524201e4d4000beadfabd4b72566b P 4dce95e2db6d454f9a636ef4f25cd23b: No bootstrap required, opened a new log
I20260812 05:20:26.371841 3452 ts_tablet_manager.cc:1397] T dfc524201e4d4000beadfabd4b72566b P 4dce95e2db6d454f9a636ef4f25cd23b: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260812 05:20:26.371958 3452 raft_consensus.cc:348] T dfc524201e4d4000beadfabd4b72566b P 4dce95e2db6d454f9a636ef4f25cd23b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dce95e2db6d454f9a636ef4f25cd23b" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 46417 } } peers { permanent_uuid: "5b0b9ad99299469c8850bf5f32f62180" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 33141 } } peers { permanent_uuid: "04f98f06eae54d60b1f5bdb390d4b9ef" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 37167 } }
I20260812 05:20:26.372008 3452 raft_consensus.cc:374] T dfc524201e4d4000beadfabd4b72566b P 4dce95e2db6d454f9a636ef4f25cd23b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 05:20:26.372020 3452 raft_consensus.cc:729] T dfc524201e4d4000beadfabd4b72566b P 4dce95e2db6d454f9a636ef4f25cd23b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4dce95e2db6d454f9a636ef4f25cd23b, State: Initialized, Role: FOLLOWER
I20260812 05:20:26.372053 3452 consensus_queue.cc:260] T dfc524201e4d4000beadfabd4b72566b P 4dce95e2db6d454f9a636ef4f25cd23b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dce95e2db6d454f9a636ef4f25cd23b" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 46417 } } peers { permanent_uuid: "5b0b9ad99299469c8850bf5f32f62180" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 33141 } } peers { permanent_uuid: "04f98f06eae54d60b1f5bdb390d4b9ef" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 37167 } }
I20260812 05:20:26.372148 3452 ts_tablet_manager.cc:1428] T dfc524201e4d4000beadfabd4b72566b P 4dce95e2db6d454f9a636ef4f25cd23b: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260812 05:20:26.372207 3459 tablet_replica.cc:439] T dfc524201e4d4000beadfabd4b72566b P 4dce95e2db6d454f9a636ef4f25cd23b: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dce95e2db6d454f9a636ef4f25cd23b" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 46417 } } peers { permanent_uuid: "5b0b9ad99299469c8850bf5f32f62180" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 33141 } } peers { permanent_uuid: "04f98f06eae54d60b1f5bdb390d4b9ef" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 37167 } } }
I20260812 05:20:26.372305 3459 tablet_replica.cc:442] T dfc524201e4d4000beadfabd4b72566b P 4dce95e2db6d454f9a636ef4f25cd23b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 05:20:26.372499 3450 tablet_bootstrap.cc:492] T dfc524201e4d4000beadfabd4b72566b P 04f98f06eae54d60b1f5bdb390d4b9ef: No bootstrap required, opened a new log
I20260812 05:20:26.372552 3450 ts_tablet_manager.cc:1397] T dfc524201e4d4000beadfabd4b72566b P 04f98f06eae54d60b1f5bdb390d4b9ef: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260812 05:20:26.372690 3450 raft_consensus.cc:348] T dfc524201e4d4000beadfabd4b72566b P 04f98f06eae54d60b1f5bdb390d4b9ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dce95e2db6d454f9a636ef4f25cd23b" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 46417 } } peers { permanent_uuid: "5b0b9ad99299469c8850bf5f32f62180" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 33141 } } peers { permanent_uuid: "04f98f06eae54d60b1f5bdb390d4b9ef" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 37167 } }
I20260812 05:20:26.372768 3450 raft_consensus.cc:374] T dfc524201e4d4000beadfabd4b72566b P 04f98f06eae54d60b1f5bdb390d4b9ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 05:20:26.372784 3450 raft_consensus.cc:729] T dfc524201e4d4000beadfabd4b72566b P 04f98f06eae54d60b1f5bdb390d4b9ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04f98f06eae54d60b1f5bdb390d4b9ef, State: Initialized, Role: FOLLOWER
I20260812 05:20:26.372823 3450 consensus_queue.cc:260] T dfc524201e4d4000beadfabd4b72566b P 04f98f06eae54d60b1f5bdb390d4b9ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dce95e2db6d454f9a636ef4f25cd23b" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 46417 } } peers { permanent_uuid: "5b0b9ad99299469c8850bf5f32f62180" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 33141 } } peers { permanent_uuid: "04f98f06eae54d60b1f5bdb390d4b9ef" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 37167 } }
I20260812 05:20:26.372933 3450 ts_tablet_manager.cc:1428] T dfc524201e4d4000beadfabd4b72566b P 04f98f06eae54d60b1f5bdb390d4b9ef: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260812 05:20:26.373052 3456 tablet_replica.cc:439] T dfc524201e4d4000beadfabd4b72566b P 04f98f06eae54d60b1f5bdb390d4b9ef: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dce95e2db6d454f9a636ef4f25cd23b" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 46417 } } peers { permanent_uuid: "5b0b9ad99299469c8850bf5f32f62180" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 33141 } } peers { permanent_uuid: "04f98f06eae54d60b1f5bdb390d4b9ef" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 37167 } } }
I20260812 05:20:26.373121 3456 tablet_replica.cc:442] T dfc524201e4d4000beadfabd4b72566b P 04f98f06eae54d60b1f5bdb390d4b9ef: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 05:20:26.415992 3458 raft_consensus.cc:482] T dfc524201e4d4000beadfabd4b72566b P 5b0b9ad99299469c8850bf5f32f62180 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 05:20:26.416074 3458 raft_consensus.cc:504] T dfc524201e4d4000beadfabd4b72566b P 5b0b9ad99299469c8850bf5f32f62180 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dce95e2db6d454f9a636ef4f25cd23b" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 46417 } } peers { permanent_uuid: "5b0b9ad99299469c8850bf5f32f62180" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 33141 } } peers { permanent_uuid: "04f98f06eae54d60b1f5bdb390d4b9ef" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 37167 } }
I20260812 05:20:26.416419 3458 leader_election.cc:288] T dfc524201e4d4000beadfabd4b72566b P 5b0b9ad99299469c8850bf5f32f62180 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4dce95e2db6d454f9a636ef4f25cd23b (127.31.100.1:46417), 04f98f06eae54d60b1f5bdb390d4b9ef (127.31.100.3:37167)
I20260812 05:20:26.419114 3448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:26.416646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.100.2:35381 (local address 127.31.100.1:46417)
0812 05:20:26.416691 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0812 05:20:26.416692 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0812 05:20:26.416725 (+ 33us) server_negotiation.cc:407] Connection header received
0812 05:20:26.416802 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:20:26.416802 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:20:26.416810 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:20:26.416848 (+ 38us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 05:20:26.416981 (+ 133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:26.417486 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:26.418427 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:26.418783 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:26.418845 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:26.418853 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0812 05:20:26.418995 (+ 142us) server_negotiation.cc:299] Negotiation successful
0812 05:20:26.419052 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260812 05:20:26.419132 3476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:26.416558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.1:46417 (local address 127.31.100.2:35381)
0812 05:20:26.416672 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0812 05:20:26.416680 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0812 05:20:26.416739 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:20:26.416853 (+ 114us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:20:26.416853 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:20:26.416855 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 05:20:26.416900 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:20:26.416901 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:20:26.417547 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:20:26.417548 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:20:26.418349 (+ 801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:20:26.418351 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:20:26.418857 (+ 506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:20:26.418858 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:20:26.418912 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:26.418915 (+ 3us) client_negotiation.cc:769] Sending connection context
0812 05:20:26.418959 (+ 44us) client_negotiation.cc:240] Negotiation successful
0812 05:20:26.419002 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"mutex_wait_us":2,"thread_start_us":62,"threads_started":1}
I20260812 05:20:26.419133 3446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:26.416616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.100.2:50893 (local address 127.31.100.3:37167)
0812 05:20:26.416658 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0812 05:20:26.416660 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 05:20:26.416737 (+ 77us) server_negotiation.cc:407] Connection header received
0812 05:20:26.416902 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:20:26.416903 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:20:26.416910 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:20:26.416951 (+ 41us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 05:20:26.417104 (+ 153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:26.417583 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:26.418475 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:26.418832 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:26.418901 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:26.418909 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0812 05:20:26.419028 (+ 119us) server_negotiation.cc:299] Negotiation successful
0812 05:20:26.419069 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260812 05:20:26.419150 3474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:26.416558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.3:37167 (local address 127.31.100.2:50893)
0812 05:20:26.416670 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0812 05:20:26.416680 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0812 05:20:26.416739 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:20:26.416974 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:20:26.416975 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:20:26.416977 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 05:20:26.417043 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:20:26.417044 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:20:26.417643 (+ 599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:20:26.417644 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:20:26.418399 (+ 755us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:20:26.418402 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:20:26.418904 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:20:26.418905 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:20:26.418968 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:26.418971 (+ 3us) client_negotiation.cc:769] Sending connection context
0812 05:20:26.419016 (+ 45us) client_negotiation.cc:240] Negotiation successful
0812 05:20:26.419057 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":55,"threads_started":1}
I20260812 05:20:26.419399 2878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfc524201e4d4000beadfabd4b72566b" candidate_uuid: "5b0b9ad99299469c8850bf5f32f62180" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4dce95e2db6d454f9a636ef4f25cd23b" is_pre_election: true
I20260812 05:20:26.419477 2878 raft_consensus.cc:2454] T dfc524201e4d4000beadfabd4b72566b P 4dce95e2db6d454f9a636ef4f25cd23b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5b0b9ad99299469c8850bf5f32f62180 in term 0.
I20260812 05:20:26.419615 3335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfc524201e4d4000beadfabd4b72566b" candidate_uuid: "5b0b9ad99299469c8850bf5f32f62180" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "04f98f06eae54d60b1f5bdb390d4b9ef" is_pre_election: true
I20260812 05:20:26.419719 3335 raft_consensus.cc:2454] T dfc524201e4d4000beadfabd4b72566b P 04f98f06eae54d60b1f5bdb390d4b9ef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5b0b9ad99299469c8850bf5f32f62180 in term 0.
I20260812 05:20:26.419899 3010 leader_election.cc:302] T dfc524201e4d4000beadfabd4b72566b P 5b0b9ad99299469c8850bf5f32f62180 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4dce95e2db6d454f9a636ef4f25cd23b, 5b0b9ad99299469c8850bf5f32f62180; no voters:
I20260812 05:20:26.420037 3458 raft_consensus.cc:2781] T dfc524201e4d4000beadfabd4b72566b P 5b0b9ad99299469c8850bf5f32f62180 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 05:20:26.420076 3458 raft_consensus.cc:482] T dfc524201e4d4000beadfabd4b72566b P 5b0b9ad99299469c8850bf5f32f62180 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 05:20:26.420096 3458 raft_consensus.cc:3037] T dfc524201e4d4000beadfabd4b72566b P 5b0b9ad99299469c8850bf5f32f62180 [term 0 FOLLOWER]: Advancing to term 1
I20260812 05:20:26.420740 3458 raft_consensus.cc:504] T dfc524201e4d4000beadfabd4b72566b P 5b0b9ad99299469c8850bf5f32f62180 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dce95e2db6d454f9a636ef4f25cd23b" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 46417 } } peers { permanent_uuid: "5b0b9ad99299469c8850bf5f32f62180" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 33141 } } peers { permanent_uuid: "04f98f06eae54d60b1f5bdb390d4b9ef" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 37167 } }
I20260812 05:20:26.420858 3458 leader_election.cc:288] T dfc524201e4d4000beadfabd4b72566b P 5b0b9ad99299469c8850bf5f32f62180 [CANDIDATE]: Term 1 election: Requested vote from peers 4dce95e2db6d454f9a636ef4f25cd23b (127.31.100.1:46417), 04f98f06eae54d60b1f5bdb390d4b9ef (127.31.100.3:37167)
I20260812 05:20:26.421015 3335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfc524201e4d4000beadfabd4b72566b" candidate_uuid: "5b0b9ad99299469c8850bf5f32f62180" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "04f98f06eae54d60b1f5bdb390d4b9ef"
I20260812 05:20:26.421015 2878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfc524201e4d4000beadfabd4b72566b" candidate_uuid: "5b0b9ad99299469c8850bf5f32f62180" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4dce95e2db6d454f9a636ef4f25cd23b"
I20260812 05:20:26.421090 3335 raft_consensus.cc:3037] T dfc524201e4d4000beadfabd4b72566b P 04f98f06eae54d60b1f5bdb390d4b9ef [term 0 FOLLOWER]: Advancing to term 1
I20260812 05:20:26.421090 2878 raft_consensus.cc:3037] T dfc524201e4d4000beadfabd4b72566b P 4dce95e2db6d454f9a636ef4f25cd23b [term 0 FOLLOWER]: Advancing to term 1
I20260812 05:20:26.421850 3335 raft_consensus.cc:2454] T dfc524201e4d4000beadfabd4b72566b P 04f98f06eae54d60b1f5bdb390d4b9ef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5b0b9ad99299469c8850bf5f32f62180 in term 1.
I20260812 05:20:26.421850 2878 raft_consensus.cc:2454] T dfc524201e4d4000beadfabd4b72566b P 4dce95e2db6d454f9a636ef4f25cd23b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5b0b9ad99299469c8850bf5f32f62180 in term 1.
I20260812 05:20:26.421998 3010 leader_election.cc:302] T dfc524201e4d4000beadfabd4b72566b P 5b0b9ad99299469c8850bf5f32f62180 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4dce95e2db6d454f9a636ef4f25cd23b, 5b0b9ad99299469c8850bf5f32f62180; no voters:
I20260812 05:20:26.422102 3458 raft_consensus.cc:2781] T dfc524201e4d4000beadfabd4b72566b P 5b0b9ad99299469c8850bf5f32f62180 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 05:20:26.422212 3458 raft_consensus.cc:686] T dfc524201e4d4000beadfabd4b72566b P 5b0b9ad99299469c8850bf5f32f62180 [term 1 LEADER]: Becoming Leader. State: Replica: 5b0b9ad99299469c8850bf5f32f62180, State: Running, Role: LEADER
I20260812 05:20:26.422328 3458 consensus_queue.cc:237] T dfc524201e4d4000beadfabd4b72566b P 5b0b9ad99299469c8850bf5f32f62180 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dce95e2db6d454f9a636ef4f25cd23b" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 46417 } } peers { permanent_uuid: "5b0b9ad99299469c8850bf5f32f62180" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 33141 } } peers { permanent_uuid: "04f98f06eae54d60b1f5bdb390d4b9ef" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 37167 } }
I20260812 05:20:26.422703 3480 tablet_replica.cc:439] T dfc524201e4d4000beadfabd4b72566b P 5b0b9ad99299469c8850bf5f32f62180: TxnStatusTablet state changed. Reason: New leader 5b0b9ad99299469c8850bf5f32f62180. Latest consensus state: current_term: 1 leader_uuid: "5b0b9ad99299469c8850bf5f32f62180" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dce95e2db6d454f9a636ef4f25cd23b" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 46417 } } peers { permanent_uuid: "5b0b9ad99299469c8850bf5f32f62180" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 33141 } } peers { permanent_uuid: "04f98f06eae54d60b1f5bdb390d4b9ef" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 37167 } } }
I20260812 05:20:26.422775 3480 tablet_replica.cc:442] T dfc524201e4d4000beadfabd4b72566b P 5b0b9ad99299469c8850bf5f32f62180: This TxnStatusTablet replica's current role is: LEADER
I20260812 05:20:26.423012 3484 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260812 05:20:26.423032 2606 catalog_manager.cc:5495] T dfc524201e4d4000beadfabd4b72566b P 5b0b9ad99299469c8850bf5f32f62180 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5b0b9ad99299469c8850bf5f32f62180 (127.31.100.2). New cstate: current_term: 1 leader_uuid: "5b0b9ad99299469c8850bf5f32f62180" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dce95e2db6d454f9a636ef4f25cd23b" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 46417 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5b0b9ad99299469c8850bf5f32f62180" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 33141 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "04f98f06eae54d60b1f5bdb390d4b9ef" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 37167 } health_report { overall_health: UNKNOWN } } }
I20260812 05:20:26.423169 3484 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260812 05:20:26.429994 3468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:26.427357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.2:33141 (local address 127.0.0.1:54220)
0812 05:20:26.427426 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0812 05:20:26.427437 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0812 05:20:26.427479 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:20:26.427636 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:20:26.427637 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:20:26.427637 (+ 0us) client_negotiation.cc:189] Negotiated authn=TOKEN
0812 05:20:26.427676 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:20:26.427679 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:20:26.428497 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:20:26.428498 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:20:26.429031 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:20:26.429032 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:20:26.429476 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:20:26.429477 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:20:26.429532 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:26.429545 (+ 13us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0812 05:20:26.429826 (+ 281us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0812 05:20:26.429839 (+ 13us) client_negotiation.cc:769] Sending connection context
0812 05:20:26.429898 (+ 59us) client_negotiation.cc:240] Negotiation successful
0812 05:20:26.429941 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260812 05:20:26.429996 3444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:26.427409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54220 (local address 127.31.100.2:33141)
0812 05:20:26.427446 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0812 05:20:26.427455 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 05:20:26.427464 (+ 9us) server_negotiation.cc:407] Connection header received
0812 05:20:26.427542 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:20:26.427543 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:20:26.427560 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:20:26.427637 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 05:20:26.427713 (+ 76us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:26.428417 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:26.429105 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:26.429383 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:26.429483 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:26.429624 (+ 141us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 05:20:26.429736 (+ 112us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 05:20:26.429818 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0812 05:20:26.429913 (+ 95us) server_negotiation.cc:299] Negotiation successful
0812 05:20:26.429933 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260812 05:20:26.431542 2878 raft_consensus.cc:1261] T dfc524201e4d4000beadfabd4b72566b P 4dce95e2db6d454f9a636ef4f25cd23b [term 1 FOLLOWER]: Refusing update from remote peer 5b0b9ad99299469c8850bf5f32f62180: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 05:20:26.431813 3335 raft_consensus.cc:1261] T dfc524201e4d4000beadfabd4b72566b P 04f98f06eae54d60b1f5bdb390d4b9ef [term 1 FOLLOWER]: Refusing update from remote peer 5b0b9ad99299469c8850bf5f32f62180: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 05:20:26.431850 3480 consensus_queue.cc:1035] T dfc524201e4d4000beadfabd4b72566b P 5b0b9ad99299469c8850bf5f32f62180 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4dce95e2db6d454f9a636ef4f25cd23b" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 46417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 05:20:26.432042 3458 consensus_queue.cc:1035] T dfc524201e4d4000beadfabd4b72566b P 5b0b9ad99299469c8850bf5f32f62180 [LEADER]: Connected to new peer: Peer: permanent_uuid: "04f98f06eae54d60b1f5bdb390d4b9ef" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 37167 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 05:20:26.433099 3456 tablet_replica.cc:439] T dfc524201e4d4000beadfabd4b72566b P 04f98f06eae54d60b1f5bdb390d4b9ef: TxnStatusTablet state changed. Reason: New leader 5b0b9ad99299469c8850bf5f32f62180. Latest consensus state: current_term: 1 leader_uuid: "5b0b9ad99299469c8850bf5f32f62180" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dce95e2db6d454f9a636ef4f25cd23b" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 46417 } } peers { permanent_uuid: "5b0b9ad99299469c8850bf5f32f62180" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 33141 } } peers { permanent_uuid: "04f98f06eae54d60b1f5bdb390d4b9ef" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 37167 } } }
I20260812 05:20:26.433183 3456 tablet_replica.cc:442] T dfc524201e4d4000beadfabd4b72566b P 04f98f06eae54d60b1f5bdb390d4b9ef: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 05:20:26.433174 3459 tablet_replica.cc:439] T dfc524201e4d4000beadfabd4b72566b P 4dce95e2db6d454f9a636ef4f25cd23b: TxnStatusTablet state changed. Reason: New leader 5b0b9ad99299469c8850bf5f32f62180. Latest consensus state: current_term: 1 leader_uuid: "5b0b9ad99299469c8850bf5f32f62180" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dce95e2db6d454f9a636ef4f25cd23b" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 46417 } } peers { permanent_uuid: "5b0b9ad99299469c8850bf5f32f62180" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 33141 } } peers { permanent_uuid: "04f98f06eae54d60b1f5bdb390d4b9ef" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 37167 } } }
I20260812 05:20:26.433245 3459 tablet_replica.cc:442] T dfc524201e4d4000beadfabd4b72566b P 4dce95e2db6d454f9a636ef4f25cd23b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 05:20:26.434307 3480 tablet_replica.cc:439] T dfc524201e4d4000beadfabd4b72566b P 5b0b9ad99299469c8850bf5f32f62180: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5b0b9ad99299469c8850bf5f32f62180" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dce95e2db6d454f9a636ef4f25cd23b" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 46417 } } peers { permanent_uuid: "5b0b9ad99299469c8850bf5f32f62180" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 33141 } } peers { permanent_uuid: "04f98f06eae54d60b1f5bdb390d4b9ef" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 37167 } } }
I20260812 05:20:26.434383 3480 tablet_replica.cc:442] T dfc524201e4d4000beadfabd4b72566b P 5b0b9ad99299469c8850bf5f32f62180: This TxnStatusTablet replica's current role is: LEADER
I20260812 05:20:26.434468 3480 tablet_replica.cc:439] T dfc524201e4d4000beadfabd4b72566b P 5b0b9ad99299469c8850bf5f32f62180: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5b0b9ad99299469c8850bf5f32f62180" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dce95e2db6d454f9a636ef4f25cd23b" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 46417 } } peers { permanent_uuid: "5b0b9ad99299469c8850bf5f32f62180" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 33141 } } peers { permanent_uuid: "04f98f06eae54d60b1f5bdb390d4b9ef" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 37167 } } }
I20260812 05:20:26.434520 3480 tablet_replica.cc:442] T dfc524201e4d4000beadfabd4b72566b P 5b0b9ad99299469c8850bf5f32f62180: This TxnStatusTablet replica's current role is: LEADER
I20260812 05:20:26.434571 3456 tablet_replica.cc:439] T dfc524201e4d4000beadfabd4b72566b P 04f98f06eae54d60b1f5bdb390d4b9ef: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5b0b9ad99299469c8850bf5f32f62180" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dce95e2db6d454f9a636ef4f25cd23b" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 46417 } } peers { permanent_uuid: "5b0b9ad99299469c8850bf5f32f62180" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 33141 } } peers { permanent_uuid: "04f98f06eae54d60b1f5bdb390d4b9ef" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 37167 } } }
I20260812 05:20:26.434643 3456 tablet_replica.cc:442] T dfc524201e4d4000beadfabd4b72566b P 04f98f06eae54d60b1f5bdb390d4b9ef: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 05:20:26.434865 3489 mvcc.cc:205] Tried to move back new op lower bound from 7317553260261515264 to 7317553260226359296. Current Snapshot: MvccSnapshot[applied={T|T < 7317553260261515264}]
I20260812 05:20:26.435122 3459 tablet_replica.cc:439] T dfc524201e4d4000beadfabd4b72566b P 4dce95e2db6d454f9a636ef4f25cd23b: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5b0b9ad99299469c8850bf5f32f62180" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dce95e2db6d454f9a636ef4f25cd23b" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 46417 } } peers { permanent_uuid: "5b0b9ad99299469c8850bf5f32f62180" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 33141 } } peers { permanent_uuid: "04f98f06eae54d60b1f5bdb390d4b9ef" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 37167 } } }
I20260812 05:20:26.435204 3459 tablet_replica.cc:442] T dfc524201e4d4000beadfabd4b72566b P 4dce95e2db6d454f9a636ef4f25cd23b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 05:20:26.435366 3488 mvcc.cc:205] Tried to move back new op lower bound from 7317553260261515264 to 7317553260226359296. Current Snapshot: MvccSnapshot[applied={T|T < 7317553260261515264}]
I20260812 05:20:26.439926 3446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:26.437384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39430 (local address 127.31.100.3:37167)
0812 05:20:26.437422 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0812 05:20:26.437423 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0812 05:20:26.437499 (+ 76us) server_negotiation.cc:407] Connection header received
0812 05:20:26.437575 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:20:26.437575 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:20:26.437589 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:20:26.437639 (+ 50us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 05:20:26.437746 (+ 107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:26.438183 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:26.439234 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:26.439434 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:26.439509 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:26.439651 (+ 142us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 05:20:26.439763 (+ 112us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 05:20:26.439817 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0812 05:20:26.439865 (+ 48us) server_negotiation.cc:299] Negotiation successful
0812 05:20:26.439882 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260812 05:20:26.444335 3444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:26.442271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.100.3:59821 (local address 127.31.100.2:33141)
0812 05:20:26.442315 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0812 05:20:26.442317 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 05:20:26.442325 (+ 8us) server_negotiation.cc:407] Connection header received
0812 05:20:26.442384 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:20:26.442385 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:20:26.442393 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:20:26.442442 (+ 49us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 05:20:26.442553 (+ 111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:26.442994 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:26.443696 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:26.443894 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:26.443948 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:26.444120 (+ 172us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 05:20:26.444153 (+ 33us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 05:20:26.444192 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0812 05:20:26.444250 (+ 58us) server_negotiation.cc:299] Negotiation successful
0812 05:20:26.444271 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260812 05:20:26.444351 3430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:26.442220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.2:33141 (local address 127.31.100.3:59821)
0812 05:20:26.442279 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0812 05:20:26.442289 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0812 05:20:26.442330 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:20:26.442446 (+ 116us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:20:26.442447 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:20:26.442448 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0812 05:20:26.442499 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:20:26.442500 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:20:26.443065 (+ 565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:20:26.443067 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:20:26.443618 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:20:26.443620 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:20:26.443954 (+ 334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:20:26.443955 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:20:26.444028 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:26.444042 (+ 14us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0812 05:20:26.444199 (+ 157us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0812 05:20:26.444201 (+ 2us) client_negotiation.cc:769] Sending connection context
0812 05:20:26.444247 (+ 46us) client_negotiation.cc:240] Negotiation successful
0812 05:20:26.444290 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260812 05:20:26.445873 3490 mvcc.cc:205] Tried to move back new op lower bound from 7317553260261515264 to 7317553260226359296. Current Snapshot: MvccSnapshot[applied={T|T < 7317553260261515264}]
I20260812 05:20:26.446659 2878 raft_consensus.cc:1261] T 37bc9f2b7a4f4724a0d47e6e11ea2053 P 4dce95e2db6d454f9a636ef4f25cd23b [term 1 FOLLOWER]: Refusing update from remote peer 04f98f06eae54d60b1f5bdb390d4b9ef: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 05:20:26.446702 3115 raft_consensus.cc:1261] T 37bc9f2b7a4f4724a0d47e6e11ea2053 P 5b0b9ad99299469c8850bf5f32f62180 [term 1 FOLLOWER]: Refusing update from remote peer 04f98f06eae54d60b1f5bdb390d4b9ef: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 05:20:26.446889 3461 consensus_queue.cc:1035] T 37bc9f2b7a4f4724a0d47e6e11ea2053 P 04f98f06eae54d60b1f5bdb390d4b9ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "4dce95e2db6d454f9a636ef4f25cd23b" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 46417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 05:20:26.447034 3456 consensus_queue.cc:1035] T 37bc9f2b7a4f4724a0d47e6e11ea2053 P 04f98f06eae54d60b1f5bdb390d4b9ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b0b9ad99299469c8850bf5f32f62180" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 33141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 05:20:26.455271 3446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:26.452872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.100.2:33731 (local address 127.31.100.3:37167)
0812 05:20:26.452910 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0812 05:20:26.452912 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 05:20:26.453005 (+ 93us) server_negotiation.cc:407] Connection header received
0812 05:20:26.453105 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:20:26.453106 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:20:26.453118 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:20:26.453166 (+ 48us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 05:20:26.453318 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:26.453810 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:26.454533 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:26.454778 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:26.454842 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:26.455080 (+ 238us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 05:20:26.455106 (+ 26us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 05:20:26.455143 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0812 05:20:26.455208 (+ 65us) server_negotiation.cc:299] Negotiation successful
0812 05:20:26.455223 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260812 05:20:26.455343 3512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:26.452797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.3:37167 (local address 127.31.100.2:33731)
0812 05:20:26.452939 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0812 05:20:26.452950 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0812 05:20:26.453029 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:20:26.453208 (+ 179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:20:26.453209 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:20:26.453210 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0812 05:20:26.453259 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:20:26.453260 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:20:26.453876 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:20:26.453877 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:20:26.454462 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:20:26.454464 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:20:26.454870 (+ 406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:20:26.454871 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:20:26.454981 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:26.454997 (+ 16us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0812 05:20:26.455156 (+ 159us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0812 05:20:26.455158 (+ 2us) client_negotiation.cc:769] Sending connection context
0812 05:20:26.455206 (+ 48us) client_negotiation.cc:240] Negotiation successful
0812 05:20:26.455246 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":84,"threads_started":1}
I20260812 05:20:26.462024 2607 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44252:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 05:20:26.462117 2607 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44252:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 05:20:26.462865 2607 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 47f84840accd4b84836532f25b4a420e: Sending DeleteTablet for 3 replicas of tablet 37bc9f2b7a4f4724a0d47e6e11ea2053
I20260812 05:20:26.463198 2850 tablet_service.cc:1514] Processing DeleteTablet for tablet 37bc9f2b7a4f4724a0d47e6e11ea2053 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 05:20:26 UTC) from {username='kudu', principal='kudu/127.31.100.62@KRBTEST.COM'} at 127.0.0.1:41550
I20260812 05:20:26.463258 3086 tablet_service.cc:1514] Processing DeleteTablet for tablet 37bc9f2b7a4f4724a0d47e6e11ea2053 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 05:20:26 UTC) from {username='kudu', principal='kudu/127.31.100.62@KRBTEST.COM'} at 127.0.0.1:54216
I20260812 05:20:26.463420 3303 tablet_service.cc:1514] Processing DeleteTablet for tablet 37bc9f2b7a4f4724a0d47e6e11ea2053 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 05:20:26 UTC) from {username='kudu', principal='kudu/127.31.100.62@KRBTEST.COM'} at 127.0.0.1:39414
I20260812 05:20:26.463467 3519 tablet_replica.cc:330] T 37bc9f2b7a4f4724a0d47e6e11ea2053 P 4dce95e2db6d454f9a636ef4f25cd23b: stopping tablet replica
I20260812 05:20:26.463564 3519 raft_consensus.cc:2229] T 37bc9f2b7a4f4724a0d47e6e11ea2053 P 4dce95e2db6d454f9a636ef4f25cd23b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 05:20:26.463610 3519 raft_consensus.cc:2258] T 37bc9f2b7a4f4724a0d47e6e11ea2053 P 4dce95e2db6d454f9a636ef4f25cd23b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 05:20:26.463793 2607 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44252:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 05:20:26.463928 2607 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 05:20:26.464136 3521 tablet_replica.cc:330] T 37bc9f2b7a4f4724a0d47e6e11ea2053 P 04f98f06eae54d60b1f5bdb390d4b9ef: stopping tablet replica
I20260812 05:20:26.464282 3521 raft_consensus.cc:2229] T 37bc9f2b7a4f4724a0d47e6e11ea2053 P 04f98f06eae54d60b1f5bdb390d4b9ef [term 1 LEADER]: Raft consensus shutting down.
I20260812 05:20:26.464363 3519 ts_tablet_manager.cc:1905] T 37bc9f2b7a4f4724a0d47e6e11ea2053 P 4dce95e2db6d454f9a636ef4f25cd23b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 05:20:26.464468 3521 raft_consensus.cc:2258] T 37bc9f2b7a4f4724a0d47e6e11ea2053 P 04f98f06eae54d60b1f5bdb390d4b9ef [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 05:20:26.464938 3521 ts_tablet_manager.cc:1905] T 37bc9f2b7a4f4724a0d47e6e11ea2053 P 04f98f06eae54d60b1f5bdb390d4b9ef: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 05:20:26.465281 2850 tablet_service.cc:1467] Processing CreateTablet for tablet e975f6a9a69546e6b265de323409857b (DEFAULT_TABLE table=test-table [id=28c1e5484ec0419d8fb72a2207fc026a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 05:20:26.465422 2850 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e975f6a9a69546e6b265de323409857b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 05:20:26.466722 3303 tablet_service.cc:1467] Processing CreateTablet for tablet e975f6a9a69546e6b265de323409857b (DEFAULT_TABLE table=test-table [id=28c1e5484ec0419d8fb72a2207fc026a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 05:20:26.466837 3452 tablet_bootstrap.cc:492] T e975f6a9a69546e6b265de323409857b P 4dce95e2db6d454f9a636ef4f25cd23b: Bootstrap starting.
I20260812 05:20:26.466955 3303 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e975f6a9a69546e6b265de323409857b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 05:20:26.467213 3452 tablet_bootstrap.cc:654] T e975f6a9a69546e6b265de323409857b P 4dce95e2db6d454f9a636ef4f25cd23b: Neither blocks nor log segments found. Creating new log.
I20260812 05:20:26.467741 3086 tablet_service.cc:1467] Processing CreateTablet for tablet e975f6a9a69546e6b265de323409857b (DEFAULT_TABLE table=test-table [id=28c1e5484ec0419d8fb72a2207fc026a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 05:20:26.467841 3086 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e975f6a9a69546e6b265de323409857b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 05:20:26.468099 3450 tablet_bootstrap.cc:492] T e975f6a9a69546e6b265de323409857b P 04f98f06eae54d60b1f5bdb390d4b9ef: Bootstrap starting.
I20260812 05:20:26.468474 3450 tablet_bootstrap.cc:654] T e975f6a9a69546e6b265de323409857b P 04f98f06eae54d60b1f5bdb390d4b9ef: Neither blocks nor log segments found. Creating new log.
I20260812 05:20:26.468958 3521 ts_tablet_manager.cc:1918] T 37bc9f2b7a4f4724a0d47e6e11ea2053 P 04f98f06eae54d60b1f5bdb390d4b9ef: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260812 05:20:26.469062 3449 tablet_bootstrap.cc:492] T e975f6a9a69546e6b265de323409857b P 5b0b9ad99299469c8850bf5f32f62180: Bootstrap starting.
I20260812 05:20:26.469079 3521 log.cc:1196] T 37bc9f2b7a4f4724a0d47e6e11ea2053 P 04f98f06eae54d60b1f5bdb390d4b9ef: Deleting WAL directory at /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786512017613366-32144-0/minicluster-data/ts-2/wal/wals/37bc9f2b7a4f4724a0d47e6e11ea2053
I20260812 05:20:26.469420 3449 tablet_bootstrap.cc:654] T e975f6a9a69546e6b265de323409857b P 5b0b9ad99299469c8850bf5f32f62180: Neither blocks nor log segments found. Creating new log.
I20260812 05:20:26.469782 3521 ts_tablet_manager.cc:1939] T 37bc9f2b7a4f4724a0d47e6e11ea2053 P 04f98f06eae54d60b1f5bdb390d4b9ef: Deleting consensus metadata
I20260812 05:20:26.469458 3520 tablet_replica.cc:330] T 37bc9f2b7a4f4724a0d47e6e11ea2053 P 5b0b9ad99299469c8850bf5f32f62180: stopping tablet replica
I20260812 05:20:26.470273 2580 catalog_manager.cc:4841] TS 04f98f06eae54d60b1f5bdb390d4b9ef (127.31.100.3:37167): tablet 37bc9f2b7a4f4724a0d47e6e11ea2053 (table test-table [id=4efdb4056a394e4dbbe277f6abbf106b]) successfully deleted
I20260812 05:20:26.470312 3520 raft_consensus.cc:2229] T 37bc9f2b7a4f4724a0d47e6e11ea2053 P 5b0b9ad99299469c8850bf5f32f62180 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 05:20:26.470484 3520 raft_consensus.cc:2258] T 37bc9f2b7a4f4724a0d47e6e11ea2053 P 5b0b9ad99299469c8850bf5f32f62180 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 05:20:26.471091 3520 ts_tablet_manager.cc:1905] T 37bc9f2b7a4f4724a0d47e6e11ea2053 P 5b0b9ad99299469c8850bf5f32f62180: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 05:20:26.472621 3520 ts_tablet_manager.cc:1918] T 37bc9f2b7a4f4724a0d47e6e11ea2053 P 5b0b9ad99299469c8850bf5f32f62180: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260812 05:20:26.472748 3520 log.cc:1196] T 37bc9f2b7a4f4724a0d47e6e11ea2053 P 5b0b9ad99299469c8850bf5f32f62180: Deleting WAL directory at /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786512017613366-32144-0/minicluster-data/ts-1/wal/wals/37bc9f2b7a4f4724a0d47e6e11ea2053
I20260812 05:20:26.473076 3520 ts_tablet_manager.cc:1939] T 37bc9f2b7a4f4724a0d47e6e11ea2053 P 5b0b9ad99299469c8850bf5f32f62180: Deleting consensus metadata
I20260812 05:20:26.473305 3450 tablet_bootstrap.cc:492] T e975f6a9a69546e6b265de323409857b P 04f98f06eae54d60b1f5bdb390d4b9ef: No bootstrap required, opened a new log
I20260812 05:20:26.473347 3450 ts_tablet_manager.cc:1397] T e975f6a9a69546e6b265de323409857b P 04f98f06eae54d60b1f5bdb390d4b9ef: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20260812 05:20:26.473557 3450 raft_consensus.cc:348] T e975f6a9a69546e6b265de323409857b P 04f98f06eae54d60b1f5bdb390d4b9ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dce95e2db6d454f9a636ef4f25cd23b" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 46417 } } peers { permanent_uuid: "5b0b9ad99299469c8850bf5f32f62180" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 33141 } } peers { permanent_uuid: "04f98f06eae54d60b1f5bdb390d4b9ef" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 37167 } }
I20260812 05:20:26.473628 3450 raft_consensus.cc:374] T e975f6a9a69546e6b265de323409857b P 04f98f06eae54d60b1f5bdb390d4b9ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 05:20:26.473647 3450 raft_consensus.cc:729] T e975f6a9a69546e6b265de323409857b P 04f98f06eae54d60b1f5bdb390d4b9ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04f98f06eae54d60b1f5bdb390d4b9ef, State: Initialized, Role: FOLLOWER
I20260812 05:20:26.473690 3450 consensus_queue.cc:260] T e975f6a9a69546e6b265de323409857b P 04f98f06eae54d60b1f5bdb390d4b9ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dce95e2db6d454f9a636ef4f25cd23b" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 46417 } } peers { permanent_uuid: "5b0b9ad99299469c8850bf5f32f62180" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 33141 } } peers { permanent_uuid: "04f98f06eae54d60b1f5bdb390d4b9ef" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 37167 } }
I20260812 05:20:26.473783 3450 ts_tablet_manager.cc:1428] T e975f6a9a69546e6b265de323409857b P 04f98f06eae54d60b1f5bdb390d4b9ef: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260812 05:20:26.474036 3519 ts_tablet_manager.cc:1918] T 37bc9f2b7a4f4724a0d47e6e11ea2053 P 4dce95e2db6d454f9a636ef4f25cd23b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260812 05:20:26.474119 3519 log.cc:1196] T 37bc9f2b7a4f4724a0d47e6e11ea2053 P 4dce95e2db6d454f9a636ef4f25cd23b: Deleting WAL directory at /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786512017613366-32144-0/minicluster-data/ts-0/wal/wals/37bc9f2b7a4f4724a0d47e6e11ea2053
I20260812 05:20:26.474385 3519 ts_tablet_manager.cc:1939] T 37bc9f2b7a4f4724a0d47e6e11ea2053 P 4dce95e2db6d454f9a636ef4f25cd23b: Deleting consensus metadata
I20260812 05:20:26.474503 2584 catalog_manager.cc:4841] TS 5b0b9ad99299469c8850bf5f32f62180 (127.31.100.2:33141): tablet 37bc9f2b7a4f4724a0d47e6e11ea2053 (table test-table [id=4efdb4056a394e4dbbe277f6abbf106b]) successfully deleted
I20260812 05:20:26.474754 2587 catalog_manager.cc:4841] TS 4dce95e2db6d454f9a636ef4f25cd23b (127.31.100.1:46417): tablet 37bc9f2b7a4f4724a0d47e6e11ea2053 (table test-table [id=4efdb4056a394e4dbbe277f6abbf106b]) successfully deleted
I20260812 05:20:26.474922 3452 tablet_bootstrap.cc:492] T e975f6a9a69546e6b265de323409857b P 4dce95e2db6d454f9a636ef4f25cd23b: No bootstrap required, opened a new log
I20260812 05:20:26.474963 3452 ts_tablet_manager.cc:1397] T e975f6a9a69546e6b265de323409857b P 4dce95e2db6d454f9a636ef4f25cd23b: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.000s
I20260812 05:20:26.475100 3452 raft_consensus.cc:348] T e975f6a9a69546e6b265de323409857b P 4dce95e2db6d454f9a636ef4f25cd23b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dce95e2db6d454f9a636ef4f25cd23b" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 46417 } } peers { permanent_uuid: "5b0b9ad99299469c8850bf5f32f62180" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 33141 } } peers { permanent_uuid: "04f98f06eae54d60b1f5bdb390d4b9ef" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 37167 } }
I20260812 05:20:26.475157 3452 raft_consensus.cc:374] T e975f6a9a69546e6b265de323409857b P 4dce95e2db6d454f9a636ef4f25cd23b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 05:20:26.475174 3452 raft_consensus.cc:729] T e975f6a9a69546e6b265de323409857b P 4dce95e2db6d454f9a636ef4f25cd23b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4dce95e2db6d454f9a636ef4f25cd23b, State: Initialized, Role: FOLLOWER
I20260812 05:20:26.475214 3452 consensus_queue.cc:260] T e975f6a9a69546e6b265de323409857b P 4dce95e2db6d454f9a636ef4f25cd23b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dce95e2db6d454f9a636ef4f25cd23b" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 46417 } } peers { permanent_uuid: "5b0b9ad99299469c8850bf5f32f62180" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 33141 } } peers { permanent_uuid: "04f98f06eae54d60b1f5bdb390d4b9ef" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 37167 } }
I20260812 05:20:26.475286 3452 ts_tablet_manager.cc:1428] T e975f6a9a69546e6b265de323409857b P 4dce95e2db6d454f9a636ef4f25cd23b: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260812 05:20:26.475944 3449 tablet_bootstrap.cc:492] T e975f6a9a69546e6b265de323409857b P 5b0b9ad99299469c8850bf5f32f62180: No bootstrap required, opened a new log
I20260812 05:20:26.475991 3449 ts_tablet_manager.cc:1397] T e975f6a9a69546e6b265de323409857b P 5b0b9ad99299469c8850bf5f32f62180: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.001s
I20260812 05:20:26.476117 3449 raft_consensus.cc:348] T e975f6a9a69546e6b265de323409857b P 5b0b9ad99299469c8850bf5f32f62180 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dce95e2db6d454f9a636ef4f25cd23b" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 46417 } } peers { permanent_uuid: "5b0b9ad99299469c8850bf5f32f62180" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 33141 } } peers { permanent_uuid: "04f98f06eae54d60b1f5bdb390d4b9ef" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 37167 } }
I20260812 05:20:26.476179 3449 raft_consensus.cc:374] T e975f6a9a69546e6b265de323409857b P 5b0b9ad99299469c8850bf5f32f62180 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 05:20:26.476198 3449 raft_consensus.cc:729] T e975f6a9a69546e6b265de323409857b P 5b0b9ad99299469c8850bf5f32f62180 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b0b9ad99299469c8850bf5f32f62180, State: Initialized, Role: FOLLOWER
I20260812 05:20:26.476238 3449 consensus_queue.cc:260] T e975f6a9a69546e6b265de323409857b P 5b0b9ad99299469c8850bf5f32f62180 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dce95e2db6d454f9a636ef4f25cd23b" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 46417 } } peers { permanent_uuid: "5b0b9ad99299469c8850bf5f32f62180" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 33141 } } peers { permanent_uuid: "04f98f06eae54d60b1f5bdb390d4b9ef" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 37167 } }
I20260812 05:20:26.476325 3449 ts_tablet_manager.cc:1428] T e975f6a9a69546e6b265de323409857b P 5b0b9ad99299469c8850bf5f32f62180: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260812 05:20:26.636876 3493 raft_consensus.cc:482] T e975f6a9a69546e6b265de323409857b P 5b0b9ad99299469c8850bf5f32f62180 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 05:20:26.636986 3493 raft_consensus.cc:504] T e975f6a9a69546e6b265de323409857b P 5b0b9ad99299469c8850bf5f32f62180 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dce95e2db6d454f9a636ef4f25cd23b" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 46417 } } peers { permanent_uuid: "5b0b9ad99299469c8850bf5f32f62180" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 33141 } } peers { permanent_uuid: "04f98f06eae54d60b1f5bdb390d4b9ef" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 37167 } }
I20260812 05:20:26.637140 3493 leader_election.cc:288] T e975f6a9a69546e6b265de323409857b P 5b0b9ad99299469c8850bf5f32f62180 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4dce95e2db6d454f9a636ef4f25cd23b (127.31.100.1:46417), 04f98f06eae54d60b1f5bdb390d4b9ef (127.31.100.3:37167)
I20260812 05:20:26.637399 2878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e975f6a9a69546e6b265de323409857b" candidate_uuid: "5b0b9ad99299469c8850bf5f32f62180" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4dce95e2db6d454f9a636ef4f25cd23b" is_pre_election: true
I20260812 05:20:26.637378 3335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e975f6a9a69546e6b265de323409857b" candidate_uuid: "5b0b9ad99299469c8850bf5f32f62180" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "04f98f06eae54d60b1f5bdb390d4b9ef" is_pre_election: true
I20260812 05:20:26.637514 2878 raft_consensus.cc:2454] T e975f6a9a69546e6b265de323409857b P 4dce95e2db6d454f9a636ef4f25cd23b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5b0b9ad99299469c8850bf5f32f62180 in term 0.
I20260812 05:20:26.637518 3335 raft_consensus.cc:2454] T e975f6a9a69546e6b265de323409857b P 04f98f06eae54d60b1f5bdb390d4b9ef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5b0b9ad99299469c8850bf5f32f62180 in term 0.
I20260812 05:20:26.637682 3010 leader_election.cc:302] T e975f6a9a69546e6b265de323409857b P 5b0b9ad99299469c8850bf5f32f62180 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4dce95e2db6d454f9a636ef4f25cd23b, 5b0b9ad99299469c8850bf5f32f62180; no voters:
I20260812 05:20:26.637799 3493 raft_consensus.cc:2781] T e975f6a9a69546e6b265de323409857b P 5b0b9ad99299469c8850bf5f32f62180 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 05:20:26.637849 3493 raft_consensus.cc:482] T e975f6a9a69546e6b265de323409857b P 5b0b9ad99299469c8850bf5f32f62180 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 05:20:26.637863 3493 raft_consensus.cc:3037] T e975f6a9a69546e6b265de323409857b P 5b0b9ad99299469c8850bf5f32f62180 [term 0 FOLLOWER]: Advancing to term 1
I20260812 05:20:26.638619 3493 raft_consensus.cc:504] T e975f6a9a69546e6b265de323409857b P 5b0b9ad99299469c8850bf5f32f62180 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dce95e2db6d454f9a636ef4f25cd23b" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 46417 } } peers { permanent_uuid: "5b0b9ad99299469c8850bf5f32f62180" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 33141 } } peers { permanent_uuid: "04f98f06eae54d60b1f5bdb390d4b9ef" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 37167 } }
I20260812 05:20:26.638741 3493 leader_election.cc:288] T e975f6a9a69546e6b265de323409857b P 5b0b9ad99299469c8850bf5f32f62180 [CANDIDATE]: Term 1 election: Requested vote from peers 4dce95e2db6d454f9a636ef4f25cd23b (127.31.100.1:46417), 04f98f06eae54d60b1f5bdb390d4b9ef (127.31.100.3:37167)
I20260812 05:20:26.638896 2878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e975f6a9a69546e6b265de323409857b" candidate_uuid: "5b0b9ad99299469c8850bf5f32f62180" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4dce95e2db6d454f9a636ef4f25cd23b"
I20260812 05:20:26.638913 3335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e975f6a9a69546e6b265de323409857b" candidate_uuid: "5b0b9ad99299469c8850bf5f32f62180" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "04f98f06eae54d60b1f5bdb390d4b9ef"
I20260812 05:20:26.638964 2878 raft_consensus.cc:3037] T e975f6a9a69546e6b265de323409857b P 4dce95e2db6d454f9a636ef4f25cd23b [term 0 FOLLOWER]: Advancing to term 1
I20260812 05:20:26.638978 3335 raft_consensus.cc:3037] T e975f6a9a69546e6b265de323409857b P 04f98f06eae54d60b1f5bdb390d4b9ef [term 0 FOLLOWER]: Advancing to term 1
I20260812 05:20:26.639514 2878 raft_consensus.cc:2454] T e975f6a9a69546e6b265de323409857b P 4dce95e2db6d454f9a636ef4f25cd23b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5b0b9ad99299469c8850bf5f32f62180 in term 1.
I20260812 05:20:26.639679 3335 raft_consensus.cc:2454] T e975f6a9a69546e6b265de323409857b P 04f98f06eae54d60b1f5bdb390d4b9ef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5b0b9ad99299469c8850bf5f32f62180 in term 1.
I20260812 05:20:26.639717 3010 leader_election.cc:302] T e975f6a9a69546e6b265de323409857b P 5b0b9ad99299469c8850bf5f32f62180 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4dce95e2db6d454f9a636ef4f25cd23b, 5b0b9ad99299469c8850bf5f32f62180; no voters:
I20260812 05:20:26.639822 3493 raft_consensus.cc:2781] T e975f6a9a69546e6b265de323409857b P 5b0b9ad99299469c8850bf5f32f62180 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 05:20:26.639861 3493 raft_consensus.cc:686] T e975f6a9a69546e6b265de323409857b P 5b0b9ad99299469c8850bf5f32f62180 [term 1 LEADER]: Becoming Leader. State: Replica: 5b0b9ad99299469c8850bf5f32f62180, State: Running, Role: LEADER
I20260812 05:20:26.639909 3493 consensus_queue.cc:237] T e975f6a9a69546e6b265de323409857b P 5b0b9ad99299469c8850bf5f32f62180 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dce95e2db6d454f9a636ef4f25cd23b" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 46417 } } peers { permanent_uuid: "5b0b9ad99299469c8850bf5f32f62180" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 33141 } } peers { permanent_uuid: "04f98f06eae54d60b1f5bdb390d4b9ef" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 37167 } }
I20260812 05:20:26.640472 2607 catalog_manager.cc:5495] T e975f6a9a69546e6b265de323409857b P 5b0b9ad99299469c8850bf5f32f62180 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5b0b9ad99299469c8850bf5f32f62180 (127.31.100.2). New cstate: current_term: 1 leader_uuid: "5b0b9ad99299469c8850bf5f32f62180" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dce95e2db6d454f9a636ef4f25cd23b" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 46417 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5b0b9ad99299469c8850bf5f32f62180" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 33141 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "04f98f06eae54d60b1f5bdb390d4b9ef" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 37167 } health_report { overall_health: UNKNOWN } } }
I20260812 05:20:26.656937 3444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:26.654727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54234 (local address 127.31.100.2:33141)
0812 05:20:26.654767 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0812 05:20:26.654769 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 05:20:26.654776 (+ 7us) server_negotiation.cc:407] Connection header received
0812 05:20:26.654798 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:20:26.654798 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:20:26.654808 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:20:26.654858 (+ 50us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 05:20:26.654998 (+ 140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:26.655635 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:26.656458 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:26.656644 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:26.656691 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:26.656771 (+ 80us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 05:20:26.656798 (+ 27us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 05:20:26.656828 (+ 30us) server_negotiation.cc:1035] Waiting for connection context
0812 05:20:26.656857 (+ 29us) server_negotiation.cc:299] Negotiation successful
0812 05:20:26.656874 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260812 05:20:26.660835 3444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:26.658917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.100.2:47351 (local address 127.31.100.2:33141)
0812 05:20:26.658960 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0812 05:20:26.658961 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0812 05:20:26.658968 (+ 7us) server_negotiation.cc:407] Connection header received
0812 05:20:26.658983 (+ 15us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:20:26.658983 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:20:26.658992 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:20:26.659034 (+ 42us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 05:20:26.659142 (+ 108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:26.659615 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:26.660274 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:26.660437 (+ 163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:26.660493 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:26.660601 (+ 108us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 05:20:26.660627 (+ 26us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 05:20:26.660667 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0812 05:20:26.660722 (+ 55us) server_negotiation.cc:299] Negotiation successful
0812 05:20:26.660741 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260812 05:20:26.660836 3512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:26.658760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.2:33141 (local address 127.31.100.2:47351)
0812 05:20:26.658813 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0812 05:20:26.658823 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0812 05:20:26.658855 (+ 32us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:20:26.659041 (+ 186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:20:26.659042 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:20:26.659043 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0812 05:20:26.659096 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:20:26.659097 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:20:26.659679 (+ 582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:20:26.659680 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:20:26.660205 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:20:26.660207 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:20:26.660492 (+ 285us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:20:26.660493 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:20:26.660541 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:26.660547 (+ 6us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0812 05:20:26.660671 (+ 124us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0812 05:20:26.660672 (+ 1us) client_negotiation.cc:769] Sending connection context
0812 05:20:26.660717 (+ 45us) client_negotiation.cc:240] Negotiation successful
0812 05:20:26.660763 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260812 05:20:26.662588 3335 raft_consensus.cc:1261] T e975f6a9a69546e6b265de323409857b P 04f98f06eae54d60b1f5bdb390d4b9ef [term 1 FOLLOWER]: Refusing update from remote peer 5b0b9ad99299469c8850bf5f32f62180: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 05:20:26.662667 2878 raft_consensus.cc:1261] T e975f6a9a69546e6b265de323409857b P 4dce95e2db6d454f9a636ef4f25cd23b [term 1 FOLLOWER]: Refusing update from remote peer 5b0b9ad99299469c8850bf5f32f62180: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 05:20:26.662832 3493 consensus_queue.cc:1035] T e975f6a9a69546e6b265de323409857b P 5b0b9ad99299469c8850bf5f32f62180 [LEADER]: Connected to new peer: Peer: permanent_uuid: "04f98f06eae54d60b1f5bdb390d4b9ef" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 37167 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 05:20:26.662945 3493 consensus_queue.cc:1035] T e975f6a9a69546e6b265de323409857b P 5b0b9ad99299469c8850bf5f32f62180 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4dce95e2db6d454f9a636ef4f25cd23b" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 46417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260812 05:20:26.667904 3488 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260812 05:20:28.185702 2606 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44252:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 05:20:28.185807 2606 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44252:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 05:20:28.186766 2606 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 47f84840accd4b84836532f25b4a420e: Sending DeleteTablet for 3 replicas of tablet e975f6a9a69546e6b265de323409857b
I20260812 05:20:28.187036 3303 tablet_service.cc:1514] Processing DeleteTablet for tablet e975f6a9a69546e6b265de323409857b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 05:20:28 UTC) from {username='kudu', principal='kudu/127.31.100.62@KRBTEST.COM'} at 127.0.0.1:39414
I20260812 05:20:28.187045 2850 tablet_service.cc:1514] Processing DeleteTablet for tablet e975f6a9a69546e6b265de323409857b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 05:20:28 UTC) from {username='kudu', principal='kudu/127.31.100.62@KRBTEST.COM'} at 127.0.0.1:41550
I20260812 05:20:28.187083 3086 tablet_service.cc:1514] Processing DeleteTablet for tablet e975f6a9a69546e6b265de323409857b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 05:20:28 UTC) from {username='kudu', principal='kudu/127.31.100.62@KRBTEST.COM'} at 127.0.0.1:54216
I20260812 05:20:28.187614 3574 tablet_replica.cc:330] T e975f6a9a69546e6b265de323409857b P 4dce95e2db6d454f9a636ef4f25cd23b: stopping tablet replica
I20260812 05:20:28.187686 3574 raft_consensus.cc:2229] T e975f6a9a69546e6b265de323409857b P 4dce95e2db6d454f9a636ef4f25cd23b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 05:20:28.187727 3574 raft_consensus.cc:2258] T e975f6a9a69546e6b265de323409857b P 4dce95e2db6d454f9a636ef4f25cd23b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 05:20:28.187840 3576 tablet_replica.cc:330] T e975f6a9a69546e6b265de323409857b P 04f98f06eae54d60b1f5bdb390d4b9ef: stopping tablet replica
I20260812 05:20:28.187901 3576 raft_consensus.cc:2229] T e975f6a9a69546e6b265de323409857b P 04f98f06eae54d60b1f5bdb390d4b9ef [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 05:20:28.187939 3576 raft_consensus.cc:2258] T e975f6a9a69546e6b265de323409857b P 04f98f06eae54d60b1f5bdb390d4b9ef [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 05:20:28.188163 3574 ts_tablet_manager.cc:1905] T e975f6a9a69546e6b265de323409857b P 4dce95e2db6d454f9a636ef4f25cd23b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 05:20:28.189816 3574 ts_tablet_manager.cc:1918] T e975f6a9a69546e6b265de323409857b P 4dce95e2db6d454f9a636ef4f25cd23b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260812 05:20:28.189888 3574 log.cc:1196] T e975f6a9a69546e6b265de323409857b P 4dce95e2db6d454f9a636ef4f25cd23b: Deleting WAL directory at /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786512017613366-32144-0/minicluster-data/ts-0/wal/wals/e975f6a9a69546e6b265de323409857b
I20260812 05:20:28.190122 3574 ts_tablet_manager.cc:1939] T e975f6a9a69546e6b265de323409857b P 4dce95e2db6d454f9a636ef4f25cd23b: Deleting consensus metadata
I20260812 05:20:28.190284 3580 tablet_replica.cc:330] T e975f6a9a69546e6b265de323409857b P 5b0b9ad99299469c8850bf5f32f62180: stopping tablet replica
I20260812 05:20:28.190351 3580 raft_consensus.cc:2229] T e975f6a9a69546e6b265de323409857b P 5b0b9ad99299469c8850bf5f32f62180 [term 1 LEADER]: Raft consensus shutting down.
I20260812 05:20:28.190527 2587 catalog_manager.cc:4841] TS 4dce95e2db6d454f9a636ef4f25cd23b (127.31.100.1:46417): tablet e975f6a9a69546e6b265de323409857b (table test-table [id=28c1e5484ec0419d8fb72a2207fc026a]) successfully deleted
I20260812 05:20:28.190623 3580 raft_consensus.cc:2258] T e975f6a9a69546e6b265de323409857b P 5b0b9ad99299469c8850bf5f32f62180 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 05:20:28.190657 3576 ts_tablet_manager.cc:1905] T e975f6a9a69546e6b265de323409857b P 04f98f06eae54d60b1f5bdb390d4b9ef: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 05:20:28.190883 3580 ts_tablet_manager.cc:1905] T e975f6a9a69546e6b265de323409857b P 5b0b9ad99299469c8850bf5f32f62180: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 05:20:28.191874 3576 ts_tablet_manager.cc:1918] T e975f6a9a69546e6b265de323409857b P 04f98f06eae54d60b1f5bdb390d4b9ef: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260812 05:20:28.191980 3576 log.cc:1196] T e975f6a9a69546e6b265de323409857b P 04f98f06eae54d60b1f5bdb390d4b9ef: Deleting WAL directory at /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786512017613366-32144-0/minicluster-data/ts-2/wal/wals/e975f6a9a69546e6b265de323409857b
I20260812 05:20:28.192178 3576 ts_tablet_manager.cc:1939] T e975f6a9a69546e6b265de323409857b P 04f98f06eae54d60b1f5bdb390d4b9ef: Deleting consensus metadata
I20260812 05:20:28.192291 3580 ts_tablet_manager.cc:1918] T e975f6a9a69546e6b265de323409857b P 5b0b9ad99299469c8850bf5f32f62180: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260812 05:20:28.192348 3580 log.cc:1196] T e975f6a9a69546e6b265de323409857b P 5b0b9ad99299469c8850bf5f32f62180: Deleting WAL directory at /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786512017613366-32144-0/minicluster-data/ts-1/wal/wals/e975f6a9a69546e6b265de323409857b
I20260812 05:20:28.192471 2580 catalog_manager.cc:4841] TS 04f98f06eae54d60b1f5bdb390d4b9ef (127.31.100.3:37167): tablet e975f6a9a69546e6b265de323409857b (table test-table [id=28c1e5484ec0419d8fb72a2207fc026a]) successfully deleted
I20260812 05:20:28.192535 3580 ts_tablet_manager.cc:1939] T e975f6a9a69546e6b265de323409857b P 5b0b9ad99299469c8850bf5f32f62180: Deleting consensus metadata
I20260812 05:20:28.192795 2584 catalog_manager.cc:4841] TS 5b0b9ad99299469c8850bf5f32f62180 (127.31.100.2:33141): tablet e975f6a9a69546e6b265de323409857b (table test-table [id=28c1e5484ec0419d8fb72a2207fc026a]) successfully deleted
Aug 12 05:20:28 dist-test-slave-42z9 krb5kdc[2531](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512028, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 12 05:20:28 dist-test-slave-42z9 krb5kdc[2531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512028, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.31.100.62@KRBTEST.COM
I20260812 05:20:28.212762 3596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:28.206222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44268 (local address 127.31.100.62:45993)
0812 05:20:28.206352 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0812 05:20:28.206355 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 05:20:28.206363 (+ 8us) server_negotiation.cc:407] Connection header received
0812 05:20:28.206446 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:20:28.206447 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:20:28.206457 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:20:28.206517 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:20:28.207058 (+ 541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:28.207566 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:28.208279 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:28.208478 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:28.208532 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:28.210626 (+ 2094us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 05:20:28.210627 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 05:20:28.210628 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 05:20:28.210635 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 05:20:28.211755 (+ 1120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:20:28.212059 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:20:28.212060 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:20:28.212060 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:20:28.212114 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:20:28.212273 (+ 159us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:20:28.212274 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:20:28.212274 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:20:28.212378 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 05:20:28.212466 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0812 05:20:28.212582 (+ 116us) server_negotiation.cc:299] Negotiation successful
0812 05:20:28.212666 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100,"thread_start_us":69,"threads_started":1}
I20260812 05:20:28.214139 2606 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44268:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 05:20:28.214290 2606 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 05:20:28.215574 2850 tablet_service.cc:1467] Processing CreateTablet for tablet 1bab50ac919649ef8ecd84855da31ebc (DEFAULT_TABLE table=test-table [id=250499f055a64c1a99619d436e5987ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 05:20:28.215601 3303 tablet_service.cc:1467] Processing CreateTablet for tablet 1bab50ac919649ef8ecd84855da31ebc (DEFAULT_TABLE table=test-table [id=250499f055a64c1a99619d436e5987ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 05:20:28.215601 3086 tablet_service.cc:1467] Processing CreateTablet for tablet 1bab50ac919649ef8ecd84855da31ebc (DEFAULT_TABLE table=test-table [id=250499f055a64c1a99619d436e5987ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 05:20:28.215709 2850 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1bab50ac919649ef8ecd84855da31ebc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 05:20:28.215724 3086 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1bab50ac919649ef8ecd84855da31ebc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 05:20:28.215724 3303 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1bab50ac919649ef8ecd84855da31ebc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 05:20:28.216729 3597 tablet_bootstrap.cc:492] T 1bab50ac919649ef8ecd84855da31ebc P 4dce95e2db6d454f9a636ef4f25cd23b: Bootstrap starting.
I20260812 05:20:28.217075 3597 tablet_bootstrap.cc:654] T 1bab50ac919649ef8ecd84855da31ebc P 4dce95e2db6d454f9a636ef4f25cd23b: Neither blocks nor log segments found. Creating new log.
I20260812 05:20:28.217092 3599 tablet_bootstrap.cc:492] T 1bab50ac919649ef8ecd84855da31ebc P 5b0b9ad99299469c8850bf5f32f62180: Bootstrap starting.
I20260812 05:20:28.217365 3600 tablet_bootstrap.cc:492] T 1bab50ac919649ef8ecd84855da31ebc P 04f98f06eae54d60b1f5bdb390d4b9ef: Bootstrap starting.
I20260812 05:20:28.217501 3599 tablet_bootstrap.cc:654] T 1bab50ac919649ef8ecd84855da31ebc P 5b0b9ad99299469c8850bf5f32f62180: Neither blocks nor log segments found. Creating new log.
I20260812 05:20:28.217744 3600 tablet_bootstrap.cc:654] T 1bab50ac919649ef8ecd84855da31ebc P 04f98f06eae54d60b1f5bdb390d4b9ef: Neither blocks nor log segments found. Creating new log.
I20260812 05:20:28.218093 3599 tablet_bootstrap.cc:492] T 1bab50ac919649ef8ecd84855da31ebc P 5b0b9ad99299469c8850bf5f32f62180: No bootstrap required, opened a new log
I20260812 05:20:28.218151 3599 ts_tablet_manager.cc:1397] T 1bab50ac919649ef8ecd84855da31ebc P 5b0b9ad99299469c8850bf5f32f62180: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260812 05:20:28.218257 3600 tablet_bootstrap.cc:492] T 1bab50ac919649ef8ecd84855da31ebc P 04f98f06eae54d60b1f5bdb390d4b9ef: No bootstrap required, opened a new log
I20260812 05:20:28.218305 3600 ts_tablet_manager.cc:1397] T 1bab50ac919649ef8ecd84855da31ebc P 04f98f06eae54d60b1f5bdb390d4b9ef: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260812 05:20:28.218284 3599 raft_consensus.cc:348] T 1bab50ac919649ef8ecd84855da31ebc P 5b0b9ad99299469c8850bf5f32f62180 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dce95e2db6d454f9a636ef4f25cd23b" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 46417 } } peers { permanent_uuid: "5b0b9ad99299469c8850bf5f32f62180" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 33141 } } peers { permanent_uuid: "04f98f06eae54d60b1f5bdb390d4b9ef" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 37167 } }
I20260812 05:20:28.218362 3599 raft_consensus.cc:374] T 1bab50ac919649ef8ecd84855da31ebc P 5b0b9ad99299469c8850bf5f32f62180 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 05:20:28.218389 3599 raft_consensus.cc:729] T 1bab50ac919649ef8ecd84855da31ebc P 5b0b9ad99299469c8850bf5f32f62180 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b0b9ad99299469c8850bf5f32f62180, State: Initialized, Role: FOLLOWER
I20260812 05:20:28.218425 3600 raft_consensus.cc:348] T 1bab50ac919649ef8ecd84855da31ebc P 04f98f06eae54d60b1f5bdb390d4b9ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dce95e2db6d454f9a636ef4f25cd23b" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 46417 } } peers { permanent_uuid: "5b0b9ad99299469c8850bf5f32f62180" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 33141 } } peers { permanent_uuid: "04f98f06eae54d60b1f5bdb390d4b9ef" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 37167 } }
I20260812 05:20:28.218457 3599 consensus_queue.cc:260] T 1bab50ac919649ef8ecd84855da31ebc P 5b0b9ad99299469c8850bf5f32f62180 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dce95e2db6d454f9a636ef4f25cd23b" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 46417 } } peers { permanent_uuid: "5b0b9ad99299469c8850bf5f32f62180" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 33141 } } peers { permanent_uuid: "04f98f06eae54d60b1f5bdb390d4b9ef" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 37167 } }
I20260812 05:20:28.218492 3600 raft_consensus.cc:374] T 1bab50ac919649ef8ecd84855da31ebc P 04f98f06eae54d60b1f5bdb390d4b9ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 05:20:28.218519 3600 raft_consensus.cc:729] T 1bab50ac919649ef8ecd84855da31ebc P 04f98f06eae54d60b1f5bdb390d4b9ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04f98f06eae54d60b1f5bdb390d4b9ef, State: Initialized, Role: FOLLOWER
I20260812 05:20:28.218565 3600 consensus_queue.cc:260] T 1bab50ac919649ef8ecd84855da31ebc P 04f98f06eae54d60b1f5bdb390d4b9ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dce95e2db6d454f9a636ef4f25cd23b" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 46417 } } peers { permanent_uuid: "5b0b9ad99299469c8850bf5f32f62180" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 33141 } } peers { permanent_uuid: "04f98f06eae54d60b1f5bdb390d4b9ef" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 37167 } }
I20260812 05:20:28.218612 3599 ts_tablet_manager.cc:1428] T 1bab50ac919649ef8ecd84855da31ebc P 5b0b9ad99299469c8850bf5f32f62180: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260812 05:20:28.218730 3600 ts_tablet_manager.cc:1428] T 1bab50ac919649ef8ecd84855da31ebc P 04f98f06eae54d60b1f5bdb390d4b9ef: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260812 05:20:28.219455 3597 tablet_bootstrap.cc:492] T 1bab50ac919649ef8ecd84855da31ebc P 4dce95e2db6d454f9a636ef4f25cd23b: No bootstrap required, opened a new log
I20260812 05:20:28.219509 3597 ts_tablet_manager.cc:1397] T 1bab50ac919649ef8ecd84855da31ebc P 4dce95e2db6d454f9a636ef4f25cd23b: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260812 05:20:28.219655 3597 raft_consensus.cc:348] T 1bab50ac919649ef8ecd84855da31ebc P 4dce95e2db6d454f9a636ef4f25cd23b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dce95e2db6d454f9a636ef4f25cd23b" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 46417 } } peers { permanent_uuid: "5b0b9ad99299469c8850bf5f32f62180" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 33141 } } peers { permanent_uuid: "04f98f06eae54d60b1f5bdb390d4b9ef" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 37167 } }
I20260812 05:20:28.219720 3597 raft_consensus.cc:374] T 1bab50ac919649ef8ecd84855da31ebc P 4dce95e2db6d454f9a636ef4f25cd23b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 05:20:28.219733 3597 raft_consensus.cc:729] T 1bab50ac919649ef8ecd84855da31ebc P 4dce95e2db6d454f9a636ef4f25cd23b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4dce95e2db6d454f9a636ef4f25cd23b, State: Initialized, Role: FOLLOWER
I20260812 05:20:28.219766 3597 consensus_queue.cc:260] T 1bab50ac919649ef8ecd84855da31ebc P 4dce95e2db6d454f9a636ef4f25cd23b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dce95e2db6d454f9a636ef4f25cd23b" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 46417 } } peers { permanent_uuid: "5b0b9ad99299469c8850bf5f32f62180" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 33141 } } peers { permanent_uuid: "04f98f06eae54d60b1f5bdb390d4b9ef" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 37167 } }
I20260812 05:20:28.219916 3597 ts_tablet_manager.cc:1428] T 1bab50ac919649ef8ecd84855da31ebc P 4dce95e2db6d454f9a636ef4f25cd23b: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260812 05:20:28.261073 3608 raft_consensus.cc:482] T 1bab50ac919649ef8ecd84855da31ebc P 4dce95e2db6d454f9a636ef4f25cd23b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 05:20:28.261183 3608 raft_consensus.cc:504] T 1bab50ac919649ef8ecd84855da31ebc P 4dce95e2db6d454f9a636ef4f25cd23b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dce95e2db6d454f9a636ef4f25cd23b" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 46417 } } peers { permanent_uuid: "5b0b9ad99299469c8850bf5f32f62180" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 33141 } } peers { permanent_uuid: "04f98f06eae54d60b1f5bdb390d4b9ef" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 37167 } }
I20260812 05:20:28.261579 3608 leader_election.cc:288] T 1bab50ac919649ef8ecd84855da31ebc P 4dce95e2db6d454f9a636ef4f25cd23b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5b0b9ad99299469c8850bf5f32f62180 (127.31.100.2:33141), 04f98f06eae54d60b1f5bdb390d4b9ef (127.31.100.3:37167)
I20260812 05:20:28.265272 3611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:28.261716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.2:33141 (local address 127.31.100.1:55321)
0812 05:20:28.261862 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0812 05:20:28.261873 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0812 05:20:28.261927 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:20:28.262069 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:20:28.262070 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:20:28.262072 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 05:20:28.262139 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:20:28.262140 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:20:28.263010 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:20:28.263011 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:20:28.264280 (+ 1269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:20:28.264285 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:20:28.265029 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:20:28.265030 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:20:28.265094 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:28.265099 (+ 5us) client_negotiation.cc:769] Sending connection context
0812 05:20:28.265146 (+ 47us) client_negotiation.cc:240] Negotiation successful
0812 05:20:28.265185 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":82,"threads_started":1}
I20260812 05:20:28.265272 3612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:28.261782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.100.1:55321 (local address 127.31.100.2:33141)
0812 05:20:28.261923 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0812 05:20:28.261925 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 05:20:28.261936 (+ 11us) server_negotiation.cc:407] Connection header received
0812 05:20:28.261969 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:20:28.261970 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:20:28.261981 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:20:28.262067 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 05:20:28.262180 (+ 113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:28.262935 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:28.264382 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:28.264933 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:28.265020 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:28.265032 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0812 05:20:28.265156 (+ 124us) server_negotiation.cc:299] Negotiation successful
0812 05:20:28.265207 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":80,"threads_started":1}
I20260812 05:20:28.265843 3115 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1bab50ac919649ef8ecd84855da31ebc" candidate_uuid: "4dce95e2db6d454f9a636ef4f25cd23b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b0b9ad99299469c8850bf5f32f62180" is_pre_election: true
I20260812 05:20:28.265944 3115 raft_consensus.cc:2454] T 1bab50ac919649ef8ecd84855da31ebc P 5b0b9ad99299469c8850bf5f32f62180 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4dce95e2db6d454f9a636ef4f25cd23b in term 0.
I20260812 05:20:28.266083 3613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:28.262373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.3:37167 (local address 127.31.100.1:54131)
0812 05:20:28.262582 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0812 05:20:28.262592 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0812 05:20:28.262655 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:20:28.262796 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:20:28.262797 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:20:28.262799 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 05:20:28.262877 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:20:28.262879 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:20:28.263560 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:20:28.263561 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:20:28.265033 (+ 1472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:20:28.265035 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:20:28.265818 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:20:28.265819 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:20:28.265908 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:28.265913 (+ 5us) client_negotiation.cc:769] Sending connection context
0812 05:20:28.265965 (+ 52us) client_negotiation.cc:240] Negotiation successful
0812 05:20:28.266002 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":89,"threads_started":1}
I20260812 05:20:28.266119 2764 leader_election.cc:302] T 1bab50ac919649ef8ecd84855da31ebc P 4dce95e2db6d454f9a636ef4f25cd23b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4dce95e2db6d454f9a636ef4f25cd23b, 5b0b9ad99299469c8850bf5f32f62180; no voters:
I20260812 05:20:28.266247 3608 raft_consensus.cc:2781] T 1bab50ac919649ef8ecd84855da31ebc P 4dce95e2db6d454f9a636ef4f25cd23b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 05:20:28.266283 3608 raft_consensus.cc:482] T 1bab50ac919649ef8ecd84855da31ebc P 4dce95e2db6d454f9a636ef4f25cd23b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 05:20:28.266300 3608 raft_consensus.cc:3037] T 1bab50ac919649ef8ecd84855da31ebc P 4dce95e2db6d454f9a636ef4f25cd23b [term 0 FOLLOWER]: Advancing to term 1
I20260812 05:20:28.266323 3614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:28.262444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.100.1:54131 (local address 127.31.100.3:37167)
0812 05:20:28.262583 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0812 05:20:28.262586 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 05:20:28.262638 (+ 52us) server_negotiation.cc:407] Connection header received
0812 05:20:28.262708 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:20:28.262709 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:20:28.262719 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:20:28.262781 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 05:20:28.262950 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:28.263482 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:28.265106 (+ 1624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:28.265720 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:28.265782 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:28.265790 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0812 05:20:28.266181 (+ 391us) server_negotiation.cc:299] Negotiation successful
0812 05:20:28.266229 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105,"thread_start_us":73,"threads_started":1}
I20260812 05:20:28.266513 3335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1bab50ac919649ef8ecd84855da31ebc" candidate_uuid: "4dce95e2db6d454f9a636ef4f25cd23b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "04f98f06eae54d60b1f5bdb390d4b9ef" is_pre_election: true
I20260812 05:20:28.266582 3335 raft_consensus.cc:2454] T 1bab50ac919649ef8ecd84855da31ebc P 04f98f06eae54d60b1f5bdb390d4b9ef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4dce95e2db6d454f9a636ef4f25cd23b in term 0.
I20260812 05:20:28.267086 3608 raft_consensus.cc:504] T 1bab50ac919649ef8ecd84855da31ebc P 4dce95e2db6d454f9a636ef4f25cd23b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dce95e2db6d454f9a636ef4f25cd23b" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 46417 } } peers { permanent_uuid: "5b0b9ad99299469c8850bf5f32f62180" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 33141 } } peers { permanent_uuid: "04f98f06eae54d60b1f5bdb390d4b9ef" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 37167 } }
I20260812 05:20:28.267205 3608 leader_election.cc:288] T 1bab50ac919649ef8ecd84855da31ebc P 4dce95e2db6d454f9a636ef4f25cd23b [CANDIDATE]: Term 1 election: Requested vote from peers 5b0b9ad99299469c8850bf5f32f62180 (127.31.100.2:33141), 04f98f06eae54d60b1f5bdb390d4b9ef (127.31.100.3:37167)
I20260812 05:20:28.267366 3115 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1bab50ac919649ef8ecd84855da31ebc" candidate_uuid: "4dce95e2db6d454f9a636ef4f25cd23b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b0b9ad99299469c8850bf5f32f62180"
I20260812 05:20:28.267380 3335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1bab50ac919649ef8ecd84855da31ebc" candidate_uuid: "4dce95e2db6d454f9a636ef4f25cd23b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "04f98f06eae54d60b1f5bdb390d4b9ef"
I20260812 05:20:28.267442 3115 raft_consensus.cc:3037] T 1bab50ac919649ef8ecd84855da31ebc P 5b0b9ad99299469c8850bf5f32f62180 [term 0 FOLLOWER]: Advancing to term 1
I20260812 05:20:28.267460 3335 raft_consensus.cc:3037] T 1bab50ac919649ef8ecd84855da31ebc P 04f98f06eae54d60b1f5bdb390d4b9ef [term 0 FOLLOWER]: Advancing to term 1
I20260812 05:20:28.268023 3115 raft_consensus.cc:2454] T 1bab50ac919649ef8ecd84855da31ebc P 5b0b9ad99299469c8850bf5f32f62180 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4dce95e2db6d454f9a636ef4f25cd23b in term 1.
I20260812 05:20:28.268200 2764 leader_election.cc:302] T 1bab50ac919649ef8ecd84855da31ebc P 4dce95e2db6d454f9a636ef4f25cd23b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4dce95e2db6d454f9a636ef4f25cd23b, 5b0b9ad99299469c8850bf5f32f62180; no voters:
I20260812 05:20:28.268260 3335 raft_consensus.cc:2454] T 1bab50ac919649ef8ecd84855da31ebc P 04f98f06eae54d60b1f5bdb390d4b9ef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4dce95e2db6d454f9a636ef4f25cd23b in term 1.
I20260812 05:20:28.268290 3608 raft_consensus.cc:2781] T 1bab50ac919649ef8ecd84855da31ebc P 4dce95e2db6d454f9a636ef4f25cd23b [term 1 FOLLOWER]: Leader election won for term 1
I20260812 05:20:28.268421 3608 raft_consensus.cc:686] T 1bab50ac919649ef8ecd84855da31ebc P 4dce95e2db6d454f9a636ef4f25cd23b [term 1 LEADER]: Becoming Leader. State: Replica: 4dce95e2db6d454f9a636ef4f25cd23b, State: Running, Role: LEADER
I20260812 05:20:28.268493 3608 consensus_queue.cc:237] T 1bab50ac919649ef8ecd84855da31ebc P 4dce95e2db6d454f9a636ef4f25cd23b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dce95e2db6d454f9a636ef4f25cd23b" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 46417 } } peers { permanent_uuid: "5b0b9ad99299469c8850bf5f32f62180" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 33141 } } peers { permanent_uuid: "04f98f06eae54d60b1f5bdb390d4b9ef" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 37167 } }
I20260812 05:20:28.269222 2606 catalog_manager.cc:5495] T 1bab50ac919649ef8ecd84855da31ebc P 4dce95e2db6d454f9a636ef4f25cd23b reported cstate change: term changed from 0 to 1, leader changed from <none> to 4dce95e2db6d454f9a636ef4f25cd23b (127.31.100.1). New cstate: current_term: 1 leader_uuid: "4dce95e2db6d454f9a636ef4f25cd23b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dce95e2db6d454f9a636ef4f25cd23b" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 46417 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5b0b9ad99299469c8850bf5f32f62180" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 33141 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "04f98f06eae54d60b1f5bdb390d4b9ef" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 37167 } health_report { overall_health: UNKNOWN } } }
I20260812 05:20:28.279347 3620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:28.277083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41560 (local address 127.31.100.1:46417)
0812 05:20:28.277209 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0812 05:20:28.277211 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 05:20:28.277219 (+ 8us) server_negotiation.cc:407] Connection header received
0812 05:20:28.277252 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:20:28.277252 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:20:28.277269 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:20:28.277335 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 05:20:28.277473 (+ 138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:28.277961 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:28.278735 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:28.278923 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:28.278984 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:28.279072 (+ 88us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 05:20:28.279169 (+ 97us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 05:20:28.279217 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0812 05:20:28.279244 (+ 27us) server_negotiation.cc:299] Negotiation successful
0812 05:20:28.279259 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105,"thread_start_us":69,"threads_started":1}
I20260812 05:20:28.283869 3612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:28.281764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.100.1:57105 (local address 127.31.100.2:33141)
0812 05:20:28.281818 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0812 05:20:28.281820 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 05:20:28.281827 (+ 7us) server_negotiation.cc:407] Connection header received
0812 05:20:28.281885 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:20:28.281886 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:20:28.281896 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:20:28.281936 (+ 40us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 05:20:28.282059 (+ 123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:28.282529 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:28.283255 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:28.283448 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:28.283500 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:28.283651 (+ 151us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 05:20:28.283680 (+ 29us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 05:20:28.283720 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0812 05:20:28.283780 (+ 60us) server_negotiation.cc:299] Negotiation successful
0812 05:20:28.283799 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260812 05:20:28.283912 3626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:28.281633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.2:33141 (local address 127.31.100.1:57105)
0812 05:20:28.281779 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0812 05:20:28.281788 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0812 05:20:28.281840 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:20:28.281953 (+ 113us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:20:28.281954 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:20:28.281955 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0812 05:20:28.282009 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:20:28.282010 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:20:28.282598 (+ 588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:20:28.282599 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:20:28.283183 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:20:28.283185 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:20:28.283505 (+ 320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:20:28.283506 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:20:28.283570 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:28.283584 (+ 14us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0812 05:20:28.283727 (+ 143us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0812 05:20:28.283729 (+ 2us) client_negotiation.cc:769] Sending connection context
0812 05:20:28.283775 (+ 46us) client_negotiation.cc:240] Negotiation successful
0812 05:20:28.283814 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":69,"threads_started":1}
I20260812 05:20:28.285667 3335 raft_consensus.cc:1261] T 1bab50ac919649ef8ecd84855da31ebc P 04f98f06eae54d60b1f5bdb390d4b9ef [term 1 FOLLOWER]: Refusing update from remote peer 4dce95e2db6d454f9a636ef4f25cd23b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 05:20:28.285676 3115 raft_consensus.cc:1261] T 1bab50ac919649ef8ecd84855da31ebc P 5b0b9ad99299469c8850bf5f32f62180 [term 1 FOLLOWER]: Refusing update from remote peer 4dce95e2db6d454f9a636ef4f25cd23b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 05:20:28.285907 3616 consensus_queue.cc:1035] T 1bab50ac919649ef8ecd84855da31ebc P 4dce95e2db6d454f9a636ef4f25cd23b [LEADER]: Connected to new peer: Peer: permanent_uuid: "04f98f06eae54d60b1f5bdb390d4b9ef" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 37167 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 05:20:28.286029 3608 consensus_queue.cc:1035] T 1bab50ac919649ef8ecd84855da31ebc P 4dce95e2db6d454f9a636ef4f25cd23b [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b0b9ad99299469c8850bf5f32f62180" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 33141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 05:20:28.292495 3620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:28.290380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.100.2:44975 (local address 127.31.100.1:46417)
0812 05:20:28.290485 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0812 05:20:28.290487 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 05:20:28.290529 (+ 42us) server_negotiation.cc:407] Connection header received
0812 05:20:28.290586 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:20:28.290587 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:20:28.290595 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:20:28.290649 (+ 54us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 05:20:28.290735 (+ 86us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:28.291166 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:28.291838 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:28.292082 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:28.292136 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:28.292278 (+ 142us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 05:20:28.292313 (+ 35us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 05:20:28.292361 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0812 05:20:28.292428 (+ 67us) server_negotiation.cc:299] Negotiation successful
0812 05:20:28.292446 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260812 05:20:28.292510 3630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:28.290350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.1:46417 (local address 127.31.100.2:44975)
0812 05:20:28.290482 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0812 05:20:28.290492 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0812 05:20:28.290542 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:20:28.290650 (+ 108us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:20:28.290651 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:20:28.290652 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0812 05:20:28.290693 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:20:28.290694 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:20:28.291230 (+ 536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:20:28.291230 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:20:28.291768 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:20:28.291769 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:20:28.292146 (+ 377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:20:28.292147 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:20:28.292206 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:28.292210 (+ 4us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0812 05:20:28.292373 (+ 163us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0812 05:20:28.292375 (+ 2us) client_negotiation.cc:769] Sending connection context
0812 05:20:28.292423 (+ 48us) client_negotiation.cc:240] Negotiation successful
0812 05:20:28.292457 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":89,"threads_started":1}
I20260812 05:20:28.298868 2606 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44268:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 05:20:28.298947 2606 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44268:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 05:20:28.299535 2606 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 47f84840accd4b84836532f25b4a420e: Sending DeleteTablet for 3 replicas of tablet 1bab50ac919649ef8ecd84855da31ebc
I20260812 05:20:28.299785 2850 tablet_service.cc:1514] Processing DeleteTablet for tablet 1bab50ac919649ef8ecd84855da31ebc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 05:20:28 UTC) from {username='kudu', principal='kudu/127.31.100.62@KRBTEST.COM'} at 127.0.0.1:41550
I20260812 05:20:28.299876 3574 tablet_replica.cc:330] T 1bab50ac919649ef8ecd84855da31ebc P 4dce95e2db6d454f9a636ef4f25cd23b: stopping tablet replica
I20260812 05:20:28.299854 3303 tablet_service.cc:1514] Processing DeleteTablet for tablet 1bab50ac919649ef8ecd84855da31ebc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 05:20:28 UTC) from {username='kudu', principal='kudu/127.31.100.62@KRBTEST.COM'} at 127.0.0.1:39414
I20260812 05:20:28.299917 3574 raft_consensus.cc:2229] T 1bab50ac919649ef8ecd84855da31ebc P 4dce95e2db6d454f9a636ef4f25cd23b [term 1 LEADER]: Raft consensus shutting down.
I20260812 05:20:28.299985 3576 tablet_replica.cc:330] T 1bab50ac919649ef8ecd84855da31ebc P 04f98f06eae54d60b1f5bdb390d4b9ef: stopping tablet replica
I20260812 05:20:28.299999 3574 raft_consensus.cc:2258] T 1bab50ac919649ef8ecd84855da31ebc P 4dce95e2db6d454f9a636ef4f25cd23b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 05:20:28.300046 3576 raft_consensus.cc:2229] T 1bab50ac919649ef8ecd84855da31ebc P 04f98f06eae54d60b1f5bdb390d4b9ef [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 05:20:28.300084 3576 raft_consensus.cc:2258] T 1bab50ac919649ef8ecd84855da31ebc P 04f98f06eae54d60b1f5bdb390d4b9ef [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 05:20:28.300076 2606 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44268:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 05:20:28.300207 2606 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 05:20:28.300351 3574 ts_tablet_manager.cc:1905] T 1bab50ac919649ef8ecd84855da31ebc P 4dce95e2db6d454f9a636ef4f25cd23b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 05:20:28.300595 3086 tablet_service.cc:1514] Processing DeleteTablet for tablet 1bab50ac919649ef8ecd84855da31ebc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 05:20:28 UTC) from {username='kudu', principal='kudu/127.31.100.62@KRBTEST.COM'} at 127.0.0.1:54216
I20260812 05:20:28.300715 3580 tablet_replica.cc:330] T 1bab50ac919649ef8ecd84855da31ebc P 5b0b9ad99299469c8850bf5f32f62180: stopping tablet replica
I20260812 05:20:28.300788 3580 raft_consensus.cc:2229] T 1bab50ac919649ef8ecd84855da31ebc P 5b0b9ad99299469c8850bf5f32f62180 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 05:20:28.300829 3580 raft_consensus.cc:2258] T 1bab50ac919649ef8ecd84855da31ebc P 5b0b9ad99299469c8850bf5f32f62180 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 05:20:28.301083 3576 ts_tablet_manager.cc:1905] T 1bab50ac919649ef8ecd84855da31ebc P 04f98f06eae54d60b1f5bdb390d4b9ef: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 05:20:28.301095 3580 ts_tablet_manager.cc:1905] T 1bab50ac919649ef8ecd84855da31ebc P 5b0b9ad99299469c8850bf5f32f62180: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 05:20:28.301926 3574 ts_tablet_manager.cc:1918] T 1bab50ac919649ef8ecd84855da31ebc P 4dce95e2db6d454f9a636ef4f25cd23b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260812 05:20:28.301996 3574 log.cc:1196] T 1bab50ac919649ef8ecd84855da31ebc P 4dce95e2db6d454f9a636ef4f25cd23b: Deleting WAL directory at /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786512017613366-32144-0/minicluster-data/ts-0/wal/wals/1bab50ac919649ef8ecd84855da31ebc
I20260812 05:20:28.302201 3574 ts_tablet_manager.cc:1939] T 1bab50ac919649ef8ecd84855da31ebc P 4dce95e2db6d454f9a636ef4f25cd23b: Deleting consensus metadata
I20260812 05:20:28.302513 2587 catalog_manager.cc:4841] TS 4dce95e2db6d454f9a636ef4f25cd23b (127.31.100.1:46417): tablet 1bab50ac919649ef8ecd84855da31ebc (table test-table [id=250499f055a64c1a99619d436e5987ac]) successfully deleted
I20260812 05:20:28.302718 3576 ts_tablet_manager.cc:1918] T 1bab50ac919649ef8ecd84855da31ebc P 04f98f06eae54d60b1f5bdb390d4b9ef: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260812 05:20:28.302731 3580 ts_tablet_manager.cc:1918] T 1bab50ac919649ef8ecd84855da31ebc P 5b0b9ad99299469c8850bf5f32f62180: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260812 05:20:28.302768 3580 log.cc:1196] T 1bab50ac919649ef8ecd84855da31ebc P 5b0b9ad99299469c8850bf5f32f62180: Deleting WAL directory at /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786512017613366-32144-0/minicluster-data/ts-1/wal/wals/1bab50ac919649ef8ecd84855da31ebc
I20260812 05:20:28.302771 3576 log.cc:1196] T 1bab50ac919649ef8ecd84855da31ebc P 04f98f06eae54d60b1f5bdb390d4b9ef: Deleting WAL directory at /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786512017613366-32144-0/minicluster-data/ts-2/wal/wals/1bab50ac919649ef8ecd84855da31ebc
I20260812 05:20:28.302901 3580 ts_tablet_manager.cc:1939] T 1bab50ac919649ef8ecd84855da31ebc P 5b0b9ad99299469c8850bf5f32f62180: Deleting consensus metadata
I20260812 05:20:28.303002 3576 ts_tablet_manager.cc:1939] T 1bab50ac919649ef8ecd84855da31ebc P 04f98f06eae54d60b1f5bdb390d4b9ef: Deleting consensus metadata
I20260812 05:20:28.303259 2584 catalog_manager.cc:4841] TS 5b0b9ad99299469c8850bf5f32f62180 (127.31.100.2:33141): tablet 1bab50ac919649ef8ecd84855da31ebc (table test-table [id=250499f055a64c1a99619d436e5987ac]) successfully deleted
I20260812 05:20:28.303726 2580 catalog_manager.cc:4841] TS 04f98f06eae54d60b1f5bdb390d4b9ef (127.31.100.3:37167): tablet 1bab50ac919649ef8ecd84855da31ebc (table test-table [id=250499f055a64c1a99619d436e5987ac]) successfully deleted
I20260812 05:20:28.304054 2850 tablet_service.cc:1467] Processing CreateTablet for tablet 88437657c85a4f2da033f0f32eb0624f (DEFAULT_TABLE table=test-table [id=36572c57a3d74081bc3fdea282016554]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 05:20:28.304062 3086 tablet_service.cc:1467] Processing CreateTablet for tablet 88437657c85a4f2da033f0f32eb0624f (DEFAULT_TABLE table=test-table [id=36572c57a3d74081bc3fdea282016554]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 05:20:28.304160 3086 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 88437657c85a4f2da033f0f32eb0624f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 05:20:28.304162 2850 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 88437657c85a4f2da033f0f32eb0624f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 05:20:28.304164 3303 tablet_service.cc:1467] Processing CreateTablet for tablet 88437657c85a4f2da033f0f32eb0624f (DEFAULT_TABLE table=test-table [id=36572c57a3d74081bc3fdea282016554]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 05:20:28.304262 3303 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 88437657c85a4f2da033f0f32eb0624f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 05:20:28.305281 3597 tablet_bootstrap.cc:492] T 88437657c85a4f2da033f0f32eb0624f P 4dce95e2db6d454f9a636ef4f25cd23b: Bootstrap starting.
I20260812 05:20:28.305296 3599 tablet_bootstrap.cc:492] T 88437657c85a4f2da033f0f32eb0624f P 5b0b9ad99299469c8850bf5f32f62180: Bootstrap starting.
I20260812 05:20:28.305394 3600 tablet_bootstrap.cc:492] T 88437657c85a4f2da033f0f32eb0624f P 04f98f06eae54d60b1f5bdb390d4b9ef: Bootstrap starting.
I20260812 05:20:28.305665 3599 tablet_bootstrap.cc:654] T 88437657c85a4f2da033f0f32eb0624f P 5b0b9ad99299469c8850bf5f32f62180: Neither blocks nor log segments found. Creating new log.
I20260812 05:20:28.305667 3597 tablet_bootstrap.cc:654] T 88437657c85a4f2da033f0f32eb0624f P 4dce95e2db6d454f9a636ef4f25cd23b: Neither blocks nor log segments found. Creating new log.
I20260812 05:20:28.305732 3600 tablet_bootstrap.cc:654] T 88437657c85a4f2da033f0f32eb0624f P 04f98f06eae54d60b1f5bdb390d4b9ef: Neither blocks nor log segments found. Creating new log.
I20260812 05:20:28.306357 3600 tablet_bootstrap.cc:492] T 88437657c85a4f2da033f0f32eb0624f P 04f98f06eae54d60b1f5bdb390d4b9ef: No bootstrap required, opened a new log
I20260812 05:20:28.306407 3600 ts_tablet_manager.cc:1397] T 88437657c85a4f2da033f0f32eb0624f P 04f98f06eae54d60b1f5bdb390d4b9ef: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260812 05:20:28.306509 3597 tablet_bootstrap.cc:492] T 88437657c85a4f2da033f0f32eb0624f P 4dce95e2db6d454f9a636ef4f25cd23b: No bootstrap required, opened a new log
I20260812 05:20:28.306504 3600 raft_consensus.cc:348] T 88437657c85a4f2da033f0f32eb0624f P 04f98f06eae54d60b1f5bdb390d4b9ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dce95e2db6d454f9a636ef4f25cd23b" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 46417 } } peers { permanent_uuid: "5b0b9ad99299469c8850bf5f32f62180" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 33141 } } peers { permanent_uuid: "04f98f06eae54d60b1f5bdb390d4b9ef" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 37167 } }
I20260812 05:20:28.306541 3599 tablet_bootstrap.cc:492] T 88437657c85a4f2da033f0f32eb0624f P 5b0b9ad99299469c8850bf5f32f62180: No bootstrap required, opened a new log
I20260812 05:20:28.306553 3597 ts_tablet_manager.cc:1397] T 88437657c85a4f2da033f0f32eb0624f P 4dce95e2db6d454f9a636ef4f25cd23b: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260812 05:20:28.306561 3600 raft_consensus.cc:374] T 88437657c85a4f2da033f0f32eb0624f P 04f98f06eae54d60b1f5bdb390d4b9ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 05:20:28.306581 3600 raft_consensus.cc:729] T 88437657c85a4f2da033f0f32eb0624f P 04f98f06eae54d60b1f5bdb390d4b9ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04f98f06eae54d60b1f5bdb390d4b9ef, State: Initialized, Role: FOLLOWER
I20260812 05:20:28.306577 3599 ts_tablet_manager.cc:1397] T 88437657c85a4f2da033f0f32eb0624f P 5b0b9ad99299469c8850bf5f32f62180: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260812 05:20:28.306622 3600 consensus_queue.cc:260] T 88437657c85a4f2da033f0f32eb0624f P 04f98f06eae54d60b1f5bdb390d4b9ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dce95e2db6d454f9a636ef4f25cd23b" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 46417 } } peers { permanent_uuid: "5b0b9ad99299469c8850bf5f32f62180" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 33141 } } peers { permanent_uuid: "04f98f06eae54d60b1f5bdb390d4b9ef" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 37167 } }
I20260812 05:20:28.306671 3599 raft_consensus.cc:348] T 88437657c85a4f2da033f0f32eb0624f P 5b0b9ad99299469c8850bf5f32f62180 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dce95e2db6d454f9a636ef4f25cd23b" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 46417 } } peers { permanent_uuid: "5b0b9ad99299469c8850bf5f32f62180" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 33141 } } peers { permanent_uuid: "04f98f06eae54d60b1f5bdb390d4b9ef" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 37167 } }
I20260812 05:20:28.306711 3600 ts_tablet_manager.cc:1428] T 88437657c85a4f2da033f0f32eb0624f P 04f98f06eae54d60b1f5bdb390d4b9ef: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260812 05:20:28.306730 3599 raft_consensus.cc:374] T 88437657c85a4f2da033f0f32eb0624f P 5b0b9ad99299469c8850bf5f32f62180 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 05:20:28.306708 3597 raft_consensus.cc:348] T 88437657c85a4f2da033f0f32eb0624f P 4dce95e2db6d454f9a636ef4f25cd23b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dce95e2db6d454f9a636ef4f25cd23b" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 46417 } } peers { permanent_uuid: "5b0b9ad99299469c8850bf5f32f62180" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 33141 } } peers { permanent_uuid: "04f98f06eae54d60b1f5bdb390d4b9ef" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 37167 } }
I20260812 05:20:28.306751 3599 raft_consensus.cc:729] T 88437657c85a4f2da033f0f32eb0624f P 5b0b9ad99299469c8850bf5f32f62180 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b0b9ad99299469c8850bf5f32f62180, State: Initialized, Role: FOLLOWER
I20260812 05:20:28.306764 3597 raft_consensus.cc:374] T 88437657c85a4f2da033f0f32eb0624f P 4dce95e2db6d454f9a636ef4f25cd23b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 05:20:28.306782 3597 raft_consensus.cc:729] T 88437657c85a4f2da033f0f32eb0624f P 4dce95e2db6d454f9a636ef4f25cd23b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4dce95e2db6d454f9a636ef4f25cd23b, State: Initialized, Role: FOLLOWER
I20260812 05:20:28.306797 3599 consensus_queue.cc:260] T 88437657c85a4f2da033f0f32eb0624f P 5b0b9ad99299469c8850bf5f32f62180 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dce95e2db6d454f9a636ef4f25cd23b" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 46417 } } peers { permanent_uuid: "5b0b9ad99299469c8850bf5f32f62180" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 33141 } } peers { permanent_uuid: "04f98f06eae54d60b1f5bdb390d4b9ef" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 37167 } }
I20260812 05:20:28.306821 3597 consensus_queue.cc:260] T 88437657c85a4f2da033f0f32eb0624f P 4dce95e2db6d454f9a636ef4f25cd23b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dce95e2db6d454f9a636ef4f25cd23b" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 46417 } } peers { permanent_uuid: "5b0b9ad99299469c8850bf5f32f62180" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 33141 } } peers { permanent_uuid: "04f98f06eae54d60b1f5bdb390d4b9ef" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 37167 } }
I20260812 05:20:28.306877 3599 ts_tablet_manager.cc:1428] T 88437657c85a4f2da033f0f32eb0624f P 5b0b9ad99299469c8850bf5f32f62180: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260812 05:20:28.306922 3597 ts_tablet_manager.cc:1428] T 88437657c85a4f2da033f0f32eb0624f P 4dce95e2db6d454f9a636ef4f25cd23b: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260812 05:20:28.454244 3607 raft_consensus.cc:482] T 88437657c85a4f2da033f0f32eb0624f P 04f98f06eae54d60b1f5bdb390d4b9ef [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 05:20:28.454348 3607 raft_consensus.cc:504] T 88437657c85a4f2da033f0f32eb0624f P 04f98f06eae54d60b1f5bdb390d4b9ef [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dce95e2db6d454f9a636ef4f25cd23b" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 46417 } } peers { permanent_uuid: "5b0b9ad99299469c8850bf5f32f62180" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 33141 } } peers { permanent_uuid: "04f98f06eae54d60b1f5bdb390d4b9ef" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 37167 } }
I20260812 05:20:28.454537 3607 leader_election.cc:288] T 88437657c85a4f2da033f0f32eb0624f P 04f98f06eae54d60b1f5bdb390d4b9ef [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4dce95e2db6d454f9a636ef4f25cd23b (127.31.100.1:46417), 5b0b9ad99299469c8850bf5f32f62180 (127.31.100.2:33141)
I20260812 05:20:28.454766 3115 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88437657c85a4f2da033f0f32eb0624f" candidate_uuid: "04f98f06eae54d60b1f5bdb390d4b9ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b0b9ad99299469c8850bf5f32f62180" is_pre_election: true
I20260812 05:20:28.454766 2878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88437657c85a4f2da033f0f32eb0624f" candidate_uuid: "04f98f06eae54d60b1f5bdb390d4b9ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4dce95e2db6d454f9a636ef4f25cd23b" is_pre_election: true
I20260812 05:20:28.454864 3115 raft_consensus.cc:2454] T 88437657c85a4f2da033f0f32eb0624f P 5b0b9ad99299469c8850bf5f32f62180 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 04f98f06eae54d60b1f5bdb390d4b9ef in term 0.
I20260812 05:20:28.454864 2878 raft_consensus.cc:2454] T 88437657c85a4f2da033f0f32eb0624f P 4dce95e2db6d454f9a636ef4f25cd23b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 04f98f06eae54d60b1f5bdb390d4b9ef in term 0.
I20260812 05:20:28.455030 3234 leader_election.cc:302] T 88437657c85a4f2da033f0f32eb0624f P 04f98f06eae54d60b1f5bdb390d4b9ef [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 04f98f06eae54d60b1f5bdb390d4b9ef, 5b0b9ad99299469c8850bf5f32f62180; no voters:
I20260812 05:20:28.455132 3607 raft_consensus.cc:2781] T 88437657c85a4f2da033f0f32eb0624f P 04f98f06eae54d60b1f5bdb390d4b9ef [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 05:20:28.455169 3607 raft_consensus.cc:482] T 88437657c85a4f2da033f0f32eb0624f P 04f98f06eae54d60b1f5bdb390d4b9ef [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 05:20:28.455210 3607 raft_consensus.cc:3037] T 88437657c85a4f2da033f0f32eb0624f P 04f98f06eae54d60b1f5bdb390d4b9ef [term 0 FOLLOWER]: Advancing to term 1
I20260812 05:20:28.455921 3607 raft_consensus.cc:504] T 88437657c85a4f2da033f0f32eb0624f P 04f98f06eae54d60b1f5bdb390d4b9ef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dce95e2db6d454f9a636ef4f25cd23b" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 46417 } } peers { permanent_uuid: "5b0b9ad99299469c8850bf5f32f62180" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 33141 } } peers { permanent_uuid: "04f98f06eae54d60b1f5bdb390d4b9ef" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 37167 } }
I20260812 05:20:28.456039 3607 leader_election.cc:288] T 88437657c85a4f2da033f0f32eb0624f P 04f98f06eae54d60b1f5bdb390d4b9ef [CANDIDATE]: Term 1 election: Requested vote from peers 4dce95e2db6d454f9a636ef4f25cd23b (127.31.100.1:46417), 5b0b9ad99299469c8850bf5f32f62180 (127.31.100.2:33141)
I20260812 05:20:28.456207 3115 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88437657c85a4f2da033f0f32eb0624f" candidate_uuid: "04f98f06eae54d60b1f5bdb390d4b9ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b0b9ad99299469c8850bf5f32f62180"
I20260812 05:20:28.456207 2878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88437657c85a4f2da033f0f32eb0624f" candidate_uuid: "04f98f06eae54d60b1f5bdb390d4b9ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4dce95e2db6d454f9a636ef4f25cd23b"
I20260812 05:20:28.456279 3115 raft_consensus.cc:3037] T 88437657c85a4f2da033f0f32eb0624f P 5b0b9ad99299469c8850bf5f32f62180 [term 0 FOLLOWER]: Advancing to term 1
I20260812 05:20:28.456279 2878 raft_consensus.cc:3037] T 88437657c85a4f2da033f0f32eb0624f P 4dce95e2db6d454f9a636ef4f25cd23b [term 0 FOLLOWER]: Advancing to term 1
I20260812 05:20:28.456840 3115 raft_consensus.cc:2454] T 88437657c85a4f2da033f0f32eb0624f P 5b0b9ad99299469c8850bf5f32f62180 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 04f98f06eae54d60b1f5bdb390d4b9ef in term 1.
I20260812 05:20:28.456897 2878 raft_consensus.cc:2454] T 88437657c85a4f2da033f0f32eb0624f P 4dce95e2db6d454f9a636ef4f25cd23b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 04f98f06eae54d60b1f5bdb390d4b9ef in term 1.
I20260812 05:20:28.457015 3234 leader_election.cc:302] T 88437657c85a4f2da033f0f32eb0624f P 04f98f06eae54d60b1f5bdb390d4b9ef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 04f98f06eae54d60b1f5bdb390d4b9ef, 5b0b9ad99299469c8850bf5f32f62180; no voters:
I20260812 05:20:28.457108 3607 raft_consensus.cc:2781] T 88437657c85a4f2da033f0f32eb0624f P 04f98f06eae54d60b1f5bdb390d4b9ef [term 1 FOLLOWER]: Leader election won for term 1
I20260812 05:20:28.457273 3607 raft_consensus.cc:686] T 88437657c85a4f2da033f0f32eb0624f P 04f98f06eae54d60b1f5bdb390d4b9ef [term 1 LEADER]: Becoming Leader. State: Replica: 04f98f06eae54d60b1f5bdb390d4b9ef, State: Running, Role: LEADER
I20260812 05:20:28.457329 3607 consensus_queue.cc:237] T 88437657c85a4f2da033f0f32eb0624f P 04f98f06eae54d60b1f5bdb390d4b9ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dce95e2db6d454f9a636ef4f25cd23b" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 46417 } } peers { permanent_uuid: "5b0b9ad99299469c8850bf5f32f62180" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 33141 } } peers { permanent_uuid: "04f98f06eae54d60b1f5bdb390d4b9ef" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 37167 } }
I20260812 05:20:28.457924 2606 catalog_manager.cc:5495] T 88437657c85a4f2da033f0f32eb0624f P 04f98f06eae54d60b1f5bdb390d4b9ef reported cstate change: term changed from 0 to 1, leader changed from <none> to 04f98f06eae54d60b1f5bdb390d4b9ef (127.31.100.3). New cstate: current_term: 1 leader_uuid: "04f98f06eae54d60b1f5bdb390d4b9ef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dce95e2db6d454f9a636ef4f25cd23b" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 46417 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5b0b9ad99299469c8850bf5f32f62180" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 33141 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "04f98f06eae54d60b1f5bdb390d4b9ef" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 37167 } health_report { overall_health: HEALTHY } } }
I20260812 05:20:28.489897 3614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:28.488005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39440 (local address 127.31.100.3:37167)
0812 05:20:28.488049 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0812 05:20:28.488051 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 05:20:28.488061 (+ 10us) server_negotiation.cc:407] Connection header received
0812 05:20:28.488090 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:20:28.488091 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:20:28.488102 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:20:28.488143 (+ 41us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 05:20:28.488241 (+ 98us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:28.488704 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:28.489354 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:28.489587 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:28.489635 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:28.489729 (+ 94us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 05:20:28.489753 (+ 24us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 05:20:28.489783 (+ 30us) server_negotiation.cc:1035] Waiting for connection context
0812 05:20:28.489813 (+ 30us) server_negotiation.cc:299] Negotiation successful
0812 05:20:28.489832 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260812 05:20:28.492040 3115 raft_consensus.cc:1261] T 88437657c85a4f2da033f0f32eb0624f P 5b0b9ad99299469c8850bf5f32f62180 [term 1 FOLLOWER]: Refusing update from remote peer 04f98f06eae54d60b1f5bdb390d4b9ef: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 05:20:28.492069 2878 raft_consensus.cc:1261] T 88437657c85a4f2da033f0f32eb0624f P 4dce95e2db6d454f9a636ef4f25cd23b [term 1 FOLLOWER]: Refusing update from remote peer 04f98f06eae54d60b1f5bdb390d4b9ef: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 05:20:28.492277 3636 consensus_queue.cc:1035] T 88437657c85a4f2da033f0f32eb0624f P 04f98f06eae54d60b1f5bdb390d4b9ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b0b9ad99299469c8850bf5f32f62180" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 33141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 05:20:28.492394 3607 consensus_queue.cc:1035] T 88437657c85a4f2da033f0f32eb0624f P 04f98f06eae54d60b1f5bdb390d4b9ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "4dce95e2db6d454f9a636ef4f25cd23b" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 46417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260812 05:20:28.496608 3546 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260812 05:20:30.012817 2607 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44268:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 05:20:30.012921 2607 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44268:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 05:20:30.013700 2607 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 47f84840accd4b84836532f25b4a420e: Sending DeleteTablet for 3 replicas of tablet 88437657c85a4f2da033f0f32eb0624f
I20260812 05:20:30.013940 2850 tablet_service.cc:1514] Processing DeleteTablet for tablet 88437657c85a4f2da033f0f32eb0624f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 05:20:30 UTC) from {username='kudu', principal='kudu/127.31.100.62@KRBTEST.COM'} at 127.0.0.1:41550
I20260812 05:20:30.013968 3086 tablet_service.cc:1514] Processing DeleteTablet for tablet 88437657c85a4f2da033f0f32eb0624f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 05:20:30 UTC) from {username='kudu', principal='kudu/127.31.100.62@KRBTEST.COM'} at 127.0.0.1:54216
I20260812 05:20:30.014125 3303 tablet_service.cc:1514] Processing DeleteTablet for tablet 88437657c85a4f2da033f0f32eb0624f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 05:20:30 UTC) from {username='kudu', principal='kudu/127.31.100.62@KRBTEST.COM'} at 127.0.0.1:39414
I20260812 05:20:30.014158 3676 tablet_replica.cc:330] T 88437657c85a4f2da033f0f32eb0624f P 4dce95e2db6d454f9a636ef4f25cd23b: stopping tablet replica
I20260812 05:20:30.014216 3676 raft_consensus.cc:2229] T 88437657c85a4f2da033f0f32eb0624f P 4dce95e2db6d454f9a636ef4f25cd23b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 05:20:30.014261 3676 raft_consensus.cc:2258] T 88437657c85a4f2da033f0f32eb0624f P 4dce95e2db6d454f9a636ef4f25cd23b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 05:20:30.014304 3681 tablet_replica.cc:330] T 88437657c85a4f2da033f0f32eb0624f P 04f98f06eae54d60b1f5bdb390d4b9ef: stopping tablet replica
I20260812 05:20:30.014356 3681 raft_consensus.cc:2229] T 88437657c85a4f2da033f0f32eb0624f P 04f98f06eae54d60b1f5bdb390d4b9ef [term 1 LEADER]: Raft consensus shutting down.
I20260812 05:20:30.014398 32144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0waFvT/build/release/bin/kudu with pid 2719
I20260812 05:20:30.014417 3681 raft_consensus.cc:2258] T 88437657c85a4f2da033f0f32eb0624f P 04f98f06eae54d60b1f5bdb390d4b9ef [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 05:20:30.015203 3681 ts_tablet_manager.cc:1905] T 88437657c85a4f2da033f0f32eb0624f P 04f98f06eae54d60b1f5bdb390d4b9ef: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 05:20:30.016441 3677 tablet_replica.cc:330] T 88437657c85a4f2da033f0f32eb0624f P 5b0b9ad99299469c8850bf5f32f62180: stopping tablet replica
I20260812 05:20:30.016510 3677 raft_consensus.cc:2229] T 88437657c85a4f2da033f0f32eb0624f P 5b0b9ad99299469c8850bf5f32f62180 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 05:20:30.016551 3677 raft_consensus.cc:2258] T 88437657c85a4f2da033f0f32eb0624f P 5b0b9ad99299469c8850bf5f32f62180 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 05:20:30.016613 3681 ts_tablet_manager.cc:1918] T 88437657c85a4f2da033f0f32eb0624f P 04f98f06eae54d60b1f5bdb390d4b9ef: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260812 05:20:30.016680 3681 log.cc:1196] T 88437657c85a4f2da033f0f32eb0624f P 04f98f06eae54d60b1f5bdb390d4b9ef: Deleting WAL directory at /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786512017613366-32144-0/minicluster-data/ts-2/wal/wals/88437657c85a4f2da033f0f32eb0624f
I20260812 05:20:30.016750 3677 ts_tablet_manager.cc:1905] T 88437657c85a4f2da033f0f32eb0624f P 5b0b9ad99299469c8850bf5f32f62180: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 05:20:30.016978 3681 ts_tablet_manager.cc:1939] T 88437657c85a4f2da033f0f32eb0624f P 04f98f06eae54d60b1f5bdb390d4b9ef: Deleting consensus metadata
I20260812 05:20:30.017289 2580 catalog_manager.cc:4841] TS 04f98f06eae54d60b1f5bdb390d4b9ef (127.31.100.3:37167): tablet 88437657c85a4f2da033f0f32eb0624f (table test-table [id=36572c57a3d74081bc3fdea282016554]) successfully deleted
I20260812 05:20:30.018085 3677 ts_tablet_manager.cc:1918] T 88437657c85a4f2da033f0f32eb0624f P 5b0b9ad99299469c8850bf5f32f62180: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260812 05:20:30.018136 3677 log.cc:1196] T 88437657c85a4f2da033f0f32eb0624f P 5b0b9ad99299469c8850bf5f32f62180: Deleting WAL directory at /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786512017613366-32144-0/minicluster-data/ts-1/wal/wals/88437657c85a4f2da033f0f32eb0624f
I20260812 05:20:30.018302 3677 ts_tablet_manager.cc:1939] T 88437657c85a4f2da033f0f32eb0624f P 5b0b9ad99299469c8850bf5f32f62180: Deleting consensus metadata
I20260812 05:20:30.019306 2584 catalog_manager.cc:4841] TS 5b0b9ad99299469c8850bf5f32f62180 (127.31.100.2:33141): tablet 88437657c85a4f2da033f0f32eb0624f (table test-table [id=36572c57a3d74081bc3fdea282016554]) successfully deleted
W20260812 05:20:30.029456 2587 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.31.100.1:46417 (error 108)
I20260812 05:20:30.029688 32144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0waFvT/build/release/bin/kudu with pid 2980
I20260812 05:20:30.033780 3684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:30.029821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.1:46417 (local address 127.0.0.1:44488)
0812 05:20:30.033675 (+ 3854us) negotiation.cc:107] Waiting for socket to connect
0812 05:20:30.033733 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.100.1:46417: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":3825,"thread_start_us":67,"threads_started":1}
W20260812 05:20:30.033967 2587 catalog_manager.cc:4584] TS 4dce95e2db6d454f9a636ef4f25cd23b (127.31.100.1:46417): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 88437657c85a4f2da033f0f32eb0624f: Network error: Client connection negotiation failed: client connection to 127.31.100.1:46417: connect: Connection refused (error 111)
I20260812 05:20:30.045974 32144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0waFvT/build/release/bin/kudu with pid 3208
I20260812 05:20:30.057605 32144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0waFvT/build/release/bin/kudu with pid 2552
2026-08-12T05:20:30Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (5909 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786512017613366-32144-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 05:20:30 dist-test-slave-42z9 krb5kdc[3689](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 05:20:30 dist-test-slave-42z9 krb5kdc[3689](info): set up 2 sockets
Aug 12 05:20:30 dist-test-slave-42z9 krb5kdc[3689](info): commencing operation
krb5kdc: starting...
W20260812 05:20:31.236693 32144 mini_kdc.cc:121] Time spent starting KDC: real 1.150s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786512017613366-32144-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786512017613366-32144-0/krb5kdc/test-user.keytab.
Aug 12 05:20:31 dist-test-slave-42z9 krb5kdc[3689](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512031, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T05:20:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T05:20:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.100.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.100.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.100.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.62.keytab.
Entry for principal kudu/127.31.100.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.62.keytab.
I20260812 05:20:31.346184 32144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0waFvT/build/release/bin/kudu
/tmp/dist-test-task0waFvT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786512017613366-32144-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786512017613366-32144-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786512017613366-32144-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786512017613366-32144-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.100.62:33135
--webserver_interface=127.31.100.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.62.keytab
--principal=kudu/127.31.100.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.100.20:36977
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.100.62:33135
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786512017613366-32144-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786512017613366-32144-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786512017613366-32144-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 05:20:31.425415 3710 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 05:20:31.425743 3710 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 05:20:31.425817 3710 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 05:20:31.427022 3710 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 05:20:31.427067 3710 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 05:20:31.427088 3710 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 05:20:31.427102 3710 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 05:20:31.427119 3710 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 05:20:31.428629 3710 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.100.20:36977
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786512017613366-32144-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786512017613366-32144-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.100.62:33135
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.62.keytab
--principal=kudu/127.31.100.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.100.62:33135
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786512017613366-32144-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.100.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3710
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786512017613366-32144-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 04:20:34 UTC on ccycloud.quasar-phqdqp.root.comops.site
build id 4581871
I20260812 05:20:31.428889 3710 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 05:20:31.429092 3710 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 05:20:31.433687 3726 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 05:20:31.433789 3722 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 05:20:31.433694 3721 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 05:20:31.433722 3710 server_base.cc:1034] running on GCE node
I20260812 05:20:31.434111 3710 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 05:20:31.434376 3710 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 05:20:31.435518 3710 hybrid_clock.cc:648] HybridClock initialized: now 1786512031435504 us; error 39 us; skew 500 ppm
Aug 12 05:20:31 dist-test-slave-42z9 krb5kdc[3689](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512031, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.100.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 05:20:31.437916 3710 init.cc:377] Logged in from keytab as kudu/127.31.100.62@KRBTEST.COM (short username kudu)
I20260812 05:20:31.438308 3710 webserver.cc:460] Webserver started at http://127.31.100.62:41861/ using document root <none> and password file <none>
I20260812 05:20:31.438509 3710 fs_manager.cc:359] Metadata directory not provided
I20260812 05:20:31.438556 3710 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 05:20:31.438676 3710 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 05:20:31.439646 3710 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786512017613366-32144-0/minicluster-data/master-0/data/instance:
uuid: "4adcc5a7a1934c18b6b9e82f5a52e788"
format_stamp: "Formatted at 2026-08-12 05:20:31 on dist-test-slave-42z9"
server_key: "25a6dabf70c3b2bf5f5eb501857c3ef2"
server_key_iv: "fe644b881466e6b98cca09d9445db20c"
server_key_version: "encryptionkey@0"
I20260812 05:20:31.440059 3710 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786512017613366-32144-0/minicluster-data/master-0/wal/instance:
uuid: "4adcc5a7a1934c18b6b9e82f5a52e788"
format_stamp: "Formatted at 2026-08-12 05:20:31 on dist-test-slave-42z9"
server_key: "25a6dabf70c3b2bf5f5eb501857c3ef2"
server_key_iv: "fe644b881466e6b98cca09d9445db20c"
server_key_version: "encryptionkey@0"
I20260812 05:20:31.441272 3710 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260812 05:20:31.441989 3737 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 05:20:31.442157 3710 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260812 05:20:31.442248 3710 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786512017613366-32144-0/minicluster-data/master-0/data,/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786512017613366-32144-0/minicluster-data/master-0/wal
uuid: "4adcc5a7a1934c18b6b9e82f5a52e788"
format_stamp: "Formatted at 2026-08-12 05:20:31 on dist-test-slave-42z9"
server_key: "25a6dabf70c3b2bf5f5eb501857c3ef2"
server_key_iv: "fe644b881466e6b98cca09d9445db20c"
server_key_version: "encryptionkey@0"
I20260812 05:20:31.442307 3710 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786512017613366-32144-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786512017613366-32144-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786512017613366-32144-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 05:20:31.451067 3710 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 05:20:31.453059 3710 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 05:20:31.453192 3710 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 05:20:31.456997 3710 rpc_server.cc:307] RPC server started. Bound to: 127.31.100.62:33135
I20260812 05:20:31.457073 3819 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.100.62:33135 every 8 connection(s)
I20260812 05:20:31.457530 3710 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786512017613366-32144-0/minicluster-data/master-0/data/info.pb
I20260812 05:20:31.458395 3822 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 05:20:31.460245 32144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0waFvT/build/release/bin/kudu as pid 3710
I20260812 05:20:31.460328 32144 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786512017613366-32144-0/minicluster-data/master-0/wal/instance
I20260812 05:20:31.460486 32144 external_mini_cluster.cc:1428] Setting key 0f8cf0955ae9989575749f2baf5614d8
I20260812 05:20:31.460817 3822 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4adcc5a7a1934c18b6b9e82f5a52e788: Bootstrap starting.
I20260812 05:20:31.461551 3822 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4adcc5a7a1934c18b6b9e82f5a52e788: Neither blocks nor log segments found. Creating new log.
I20260812 05:20:31.461876 3822 log.cc:824] T 00000000000000000000000000000000 P 4adcc5a7a1934c18b6b9e82f5a52e788: Log is configured to *not* fsync() on all Append() calls
I20260812 05:20:31.462522 3822 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4adcc5a7a1934c18b6b9e82f5a52e788: No bootstrap required, opened a new log
I20260812 05:20:31.464723 3822 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4adcc5a7a1934c18b6b9e82f5a52e788 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4adcc5a7a1934c18b6b9e82f5a52e788" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 33135 } }
I20260812 05:20:31.464884 3822 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4adcc5a7a1934c18b6b9e82f5a52e788 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 05:20:31.464917 3822 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4adcc5a7a1934c18b6b9e82f5a52e788 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4adcc5a7a1934c18b6b9e82f5a52e788, State: Initialized, Role: FOLLOWER
I20260812 05:20:31.465037 3822 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4adcc5a7a1934c18b6b9e82f5a52e788 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4adcc5a7a1934c18b6b9e82f5a52e788" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 33135 } }
I20260812 05:20:31.465111 3822 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4adcc5a7a1934c18b6b9e82f5a52e788 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 05:20:31.465143 3822 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4adcc5a7a1934c18b6b9e82f5a52e788 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 05:20:31.465188 3822 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4adcc5a7a1934c18b6b9e82f5a52e788 [term 0 FOLLOWER]: Advancing to term 1
I20260812 05:20:31.465778 3822 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4adcc5a7a1934c18b6b9e82f5a52e788 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4adcc5a7a1934c18b6b9e82f5a52e788" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 33135 } }
I20260812 05:20:31.465871 3822 leader_election.cc:302] T 00000000000000000000000000000000 P 4adcc5a7a1934c18b6b9e82f5a52e788 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4adcc5a7a1934c18b6b9e82f5a52e788; no voters:
I20260812 05:20:31.466032 3822 leader_election.cc:288] T 00000000000000000000000000000000 P 4adcc5a7a1934c18b6b9e82f5a52e788 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 05:20:31.466073 3829 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4adcc5a7a1934c18b6b9e82f5a52e788 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 05:20:31.466189 3829 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4adcc5a7a1934c18b6b9e82f5a52e788 [term 1 LEADER]: Becoming Leader. State: Replica: 4adcc5a7a1934c18b6b9e82f5a52e788, State: Running, Role: LEADER
I20260812 05:20:31.466271 3829 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4adcc5a7a1934c18b6b9e82f5a52e788 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4adcc5a7a1934c18b6b9e82f5a52e788" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 33135 } }
I20260812 05:20:31.466303 3822 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4adcc5a7a1934c18b6b9e82f5a52e788 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 05:20:31.466588 3829 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4adcc5a7a1934c18b6b9e82f5a52e788 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4adcc5a7a1934c18b6b9e82f5a52e788. Latest consensus state: current_term: 1 leader_uuid: "4adcc5a7a1934c18b6b9e82f5a52e788" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4adcc5a7a1934c18b6b9e82f5a52e788" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 33135 } } }
I20260812 05:20:31.466655 3829 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4adcc5a7a1934c18b6b9e82f5a52e788 [sys.catalog]: This master's current role is: LEADER
I20260812 05:20:31.466612 3831 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4adcc5a7a1934c18b6b9e82f5a52e788 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4adcc5a7a1934c18b6b9e82f5a52e788" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4adcc5a7a1934c18b6b9e82f5a52e788" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 33135 } } }
I20260812 05:20:31.466845 3831 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4adcc5a7a1934c18b6b9e82f5a52e788 [sys.catalog]: This master's current role is: LEADER
I20260812 05:20:31.466871 3838 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
Aug 12 05:20:31 dist-test-slave-42z9 krb5kdc[3689](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512031, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.100.62@KRBTEST.COM
I20260812 05:20:31.467370 3838 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 05:20:31.468938 3838 catalog_manager.cc:1347] Generated new cluster ID: 797473f7a6424947bcaab1f329a2adeb
I20260812 05:20:31.469350 3838 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 05:20:31.471159 3827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:31.460942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33204 (local address 127.31.100.62:33135)
0812 05:20:31.461107 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0812 05:20:31.461117 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 05:20:31.461151 (+ 34us) server_negotiation.cc:407] Connection header received
0812 05:20:31.461739 (+ 588us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:20:31.461750 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:20:31.461813 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:20:31.461969 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:20:31.462543 (+ 574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:31.463919 (+ 1376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:31.464630 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:31.465224 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:31.465332 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:31.467960 (+ 2628us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 05:20:31.467964 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 05:20:31.467964 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 05:20:31.467975 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0812 05:20:31.469843 (+ 1868us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:20:31.470148 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:20:31.470151 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:20:31.470151 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:20:31.470235 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:20:31.470408 (+ 173us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:20:31.470409 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:20:31.470409 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:20:31.470629 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 05:20:31.470760 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0812 05:20:31.470869 (+ 109us) server_negotiation.cc:299] Negotiation successful
0812 05:20:31.471004 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":80,"threads_started":1}
I20260812 05:20:31.498414 3838 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 05:20:31.498950 3838 catalog_manager.cc:1504] Loading token signing keys...
I20260812 05:20:31.508216 3838 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4adcc5a7a1934c18b6b9e82f5a52e788: Generated new TSK 0
I20260812 05:20:31.508381 3838 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.31.100.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.100.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.100.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.1.keytab.
Entry for principal kudu/127.31.100.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.1.keytab.
I20260812 05:20:31.532351 32144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0waFvT/build/release/bin/kudu
/tmp/dist-test-task0waFvT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786512017613366-32144-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786512017613366-32144-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786512017613366-32144-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786512017613366-32144-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.100.1:0
--local_ip_for_outbound_sockets=127.31.100.1
--webserver_interface=127.31.100.1
--webserver_port=0
--tserver_master_addrs=127.31.100.62:33135
--keytab_file=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.1.keytab
--principal=kudu/127.31.100.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.100.20:36977
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786512017613366-32144-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786512017613366-32144-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786512017613366-32144-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 05:20:31.611758 3863 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 05:20:31.611959 3863 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 05:20:31.611991 3863 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 05:20:31.613101 3863 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 05:20:31.613137 3863 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 05:20:31.613171 3863 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.100.1
I20260812 05:20:31.614560 3863 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.100.20:36977
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786512017613366-32144-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786512017613366-32144-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.1.keytab
--principal=kudu/127.31.100.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.100.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786512017613366-32144-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.100.1
--webserver_port=0
--tserver_master_addrs=127.31.100.62:33135
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3863
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.100.1
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786512017613366-32144-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 04:20:34 UTC on ccycloud.quasar-phqdqp.root.comops.site
build id 4581871
I20260812 05:20:31.614794 3863 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 05:20:31.614990 3863 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 05:20:31.619856 3876 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 05:20:31.619850 3879 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 05:20:31.619864 3875 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 05:20:31.619992 3863 server_base.cc:1034] running on GCE node
I20260812 05:20:31.620309 3863 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 05:20:31.620570 3863 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 05:20:31.621711 3863 hybrid_clock.cc:648] HybridClock initialized: now 1786512031621693 us; error 29 us; skew 500 ppm
Aug 12 05:20:31 dist-test-slave-42z9 krb5kdc[3689](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512031, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.100.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 05:20:31.624074 3863 init.cc:377] Logged in from keytab as kudu/127.31.100.1@KRBTEST.COM (short username kudu)
I20260812 05:20:31.624486 3863 webserver.cc:460] Webserver started at http://127.31.100.1:38909/ using document root <none> and password file <none>
I20260812 05:20:31.624673 3863 fs_manager.cc:359] Metadata directory not provided
I20260812 05:20:31.624715 3863 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 05:20:31.624831 3863 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 05:20:31.625841 3863 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786512017613366-32144-0/minicluster-data/ts-0/data/instance:
uuid: "06dfdb78a31243358ef76890db439638"
format_stamp: "Formatted at 2026-08-12 05:20:31 on dist-test-slave-42z9"
server_key: "6dfb3f15f3862a4ef31dad3aba26165f"
server_key_iv: "b05817692eafa6f927d4f705afca681b"
server_key_version: "encryptionkey@0"
I20260812 05:20:31.626140 3863 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786512017613366-32144-0/minicluster-data/ts-0/wal/instance:
uuid: "06dfdb78a31243358ef76890db439638"
format_stamp: "Formatted at 2026-08-12 05:20:31 on dist-test-slave-42z9"
server_key: "6dfb3f15f3862a4ef31dad3aba26165f"
server_key_iv: "b05817692eafa6f927d4f705afca681b"
server_key_version: "encryptionkey@0"
I20260812 05:20:31.627511 3863 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260812 05:20:31.628258 3899 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 05:20:31.628436 3863 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 05:20:31.628546 3863 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786512017613366-32144-0/minicluster-data/ts-0/data,/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786512017613366-32144-0/minicluster-data/ts-0/wal
uuid: "06dfdb78a31243358ef76890db439638"
format_stamp: "Formatted at 2026-08-12 05:20:31 on dist-test-slave-42z9"
server_key: "6dfb3f15f3862a4ef31dad3aba26165f"
server_key_iv: "b05817692eafa6f927d4f705afca681b"
server_key_version: "encryptionkey@0"
I20260812 05:20:31.628597 3863 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786512017613366-32144-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786512017613366-32144-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786512017613366-32144-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 05:20:31.666203 3863 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 05:20:31.668655 3863 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 05:20:31.668768 3863 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 05:20:31.668953 3863 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 05:20:31.669219 3863 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 05:20:31.669258 3863 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 05:20:31.669296 3863 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 05:20:31.669315 3863 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 05:20:31.674829 3863 rpc_server.cc:307] RPC server started. Bound to: 127.31.100.1:42085
I20260812 05:20:31.674906 4097 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.100.1:42085 every 8 connection(s)
I20260812 05:20:31.675196 3863 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786512017613366-32144-0/minicluster-data/ts-0/data/info.pb
I20260812 05:20:31.677609 32144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0waFvT/build/release/bin/kudu as pid 3863
I20260812 05:20:31.677690 32144 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786512017613366-32144-0/minicluster-data/ts-0/wal/instance
I20260812 05:20:31.677845 32144 external_mini_cluster.cc:1428] Setting key 47d1153fd9ac0064d9378710900c3c75
Aug 12 05:20:31 dist-test-slave-42z9 krb5kdc[3689](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512031, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.100.1@KRBTEST.COM for kudu/127.31.100.62@KRBTEST.COM
I20260812 05:20:31.685084 3827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:31.675901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.100.1:41813 (local address 127.31.100.62:33135)
0812 05:20:31.675947 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0812 05:20:31.675950 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 05:20:31.676162 (+ 212us) server_negotiation.cc:407] Connection header received
0812 05:20:31.676659 (+ 497us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:20:31.676660 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:20:31.676669 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:20:31.676705 (+ 36us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:20:31.677649 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:31.678289 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:31.679276 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:31.679552 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:31.679620 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:31.681962 (+ 2342us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 05:20:31.681963 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 05:20:31.681964 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 05:20:31.681971 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 05:20:31.683564 (+ 1593us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:20:31.684076 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:20:31.684077 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:20:31.684078 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:20:31.684147 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:20:31.684465 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:20:31.684466 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:20:31.684466 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:20:31.684584 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 05:20:31.684649 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0812 05:20:31.684953 (+ 304us) server_negotiation.cc:299] Negotiation successful
0812 05:20:31.685022 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260812 05:20:31.685340 4104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:31.675888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.62:33135 (local address 127.31.100.1:41813)
0812 05:20:31.676051 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0812 05:20:31.676086 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0812 05:20:31.676571 (+ 485us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:20:31.676765 (+ 194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:20:31.676768 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:20:31.677310 (+ 542us) client_negotiation.cc:189] Negotiated authn=SASL
0812 05:20:31.677562 (+ 252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:20:31.677568 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:20:31.678377 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:20:31.678378 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:20:31.679197 (+ 819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:20:31.679200 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:20:31.679644 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:20:31.679646 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:20:31.679792 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:31.680305 (+ 513us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 05:20:31.680314 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0812 05:20:31.681859 (+ 1545us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 05:20:31.683661 (+ 1802us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 05:20:31.683668 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 05:20:31.683671 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 05:20:31.683970 (+ 299us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 05:20:31.684213 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 05:20:31.684214 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 05:20:31.684214 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 05:20:31.684357 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 05:20:31.684652 (+ 295us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 05:20:31.684659 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 05:20:31.684823 (+ 164us) client_negotiation.cc:769] Sending connection context
0812 05:20:31.684940 (+ 117us) client_negotiation.cc:240] Negotiation successful
0812 05:20:31.685115 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":89,"threads_started":1}
I20260812 05:20:31.685884 4099 heartbeater.cc:347] Connected to a master server at 127.31.100.62:33135
I20260812 05:20:31.685992 4099 heartbeater.cc:464] Registering TS with master...
I20260812 05:20:31.686179 4099 heartbeater.cc:511] Master 127.31.100.62:33135 requested a full tablet report, sending...
I20260812 05:20:31.686666 3758 ts_manager.cc:194] Registered new tserver with Master: 06dfdb78a31243358ef76890db439638 (127.31.100.1:42085)
WARNING: no policy specified for kudu/127.31.100.2@KRBTEST.COM; defaulting to no policy
I20260812 05:20:31.687319 3758 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.100.1@KRBTEST.COM'} at 127.31.100.1:41813
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.100.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.100.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.2.keytab.
Entry for principal kudu/127.31.100.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.2.keytab.
I20260812 05:20:31.701408 32144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0waFvT/build/release/bin/kudu
/tmp/dist-test-task0waFvT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786512017613366-32144-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786512017613366-32144-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786512017613366-32144-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786512017613366-32144-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.100.2:0
--local_ip_for_outbound_sockets=127.31.100.2
--webserver_interface=127.31.100.2
--webserver_port=0
--tserver_master_addrs=127.31.100.62:33135
--keytab_file=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.2.keytab
--principal=kudu/127.31.100.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.100.20:36977
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786512017613366-32144-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786512017613366-32144-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786512017613366-32144-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 05:20:31.781085 4107 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 05:20:31.781265 4107 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 05:20:31.781286 4107 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 05:20:31.782445 4107 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 05:20:31.782490 4107 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 05:20:31.782516 4107 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.100.2
I20260812 05:20:31.783923 4107 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.100.20:36977
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786512017613366-32144-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786512017613366-32144-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.2.keytab
--principal=kudu/127.31.100.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.100.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786512017613366-32144-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.100.2
--webserver_port=0
--tserver_master_addrs=127.31.100.62:33135
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4107
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.100.2
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786512017613366-32144-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 04:20:34 UTC on ccycloud.quasar-phqdqp.root.comops.site
build id 4581871
I20260812 05:20:31.784137 4107 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 05:20:31.784343 4107 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 05:20:31.788967 4122 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 05:20:31.789014 4115 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 05:20:31.788975 4118 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 05:20:31.789176 4107 server_base.cc:1034] running on GCE node
I20260812 05:20:31.789368 4107 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 05:20:31.789628 4107 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 05:20:31.790771 4107 hybrid_clock.cc:648] HybridClock initialized: now 1786512031790753 us; error 34 us; skew 500 ppm
Aug 12 05:20:31 dist-test-slave-42z9 krb5kdc[3689](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512031, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.100.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 05:20:31.793133 4107 init.cc:377] Logged in from keytab as kudu/127.31.100.2@KRBTEST.COM (short username kudu)
I20260812 05:20:31.793567 4107 webserver.cc:460] Webserver started at http://127.31.100.2:34075/ using document root <none> and password file <none>
I20260812 05:20:31.793784 4107 fs_manager.cc:359] Metadata directory not provided
I20260812 05:20:31.793826 4107 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 05:20:31.793958 4107 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 05:20:31.794926 4107 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786512017613366-32144-0/minicluster-data/ts-1/data/instance:
uuid: "972d0a1fb5a147af83b2130f29f6df58"
format_stamp: "Formatted at 2026-08-12 05:20:31 on dist-test-slave-42z9"
server_key: "c86085874218ebb840652bf60510df6a"
server_key_iv: "7c9a2810729de16209acb873b2a50f2f"
server_key_version: "encryptionkey@0"
I20260812 05:20:31.795239 4107 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786512017613366-32144-0/minicluster-data/ts-1/wal/instance:
uuid: "972d0a1fb5a147af83b2130f29f6df58"
format_stamp: "Formatted at 2026-08-12 05:20:31 on dist-test-slave-42z9"
server_key: "c86085874218ebb840652bf60510df6a"
server_key_iv: "7c9a2810729de16209acb873b2a50f2f"
server_key_version: "encryptionkey@0"
I20260812 05:20:31.796412 4107 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260812 05:20:31.797027 4130 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 05:20:31.797168 4107 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260812 05:20:31.797251 4107 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786512017613366-32144-0/minicluster-data/ts-1/data,/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786512017613366-32144-0/minicluster-data/ts-1/wal
uuid: "972d0a1fb5a147af83b2130f29f6df58"
format_stamp: "Formatted at 2026-08-12 05:20:31 on dist-test-slave-42z9"
server_key: "c86085874218ebb840652bf60510df6a"
server_key_iv: "7c9a2810729de16209acb873b2a50f2f"
server_key_version: "encryptionkey@0"
I20260812 05:20:31.797302 4107 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786512017613366-32144-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786512017613366-32144-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786512017613366-32144-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 05:20:31.811015 4107 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 05:20:31.812894 4107 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 05:20:31.812996 4107 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 05:20:31.813155 4107 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 05:20:31.813386 4107 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 05:20:31.813416 4107 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 05:20:31.813433 4107 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 05:20:31.813479 4107 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 05:20:31.818750 4107 rpc_server.cc:307] RPC server started. Bound to: 127.31.100.2:44687
I20260812 05:20:31.818799 4319 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.100.2:44687 every 8 connection(s)
I20260812 05:20:31.819063 4107 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786512017613366-32144-0/minicluster-data/ts-1/data/info.pb
Aug 12 05:20:31 dist-test-slave-42z9 krb5kdc[3689](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512031, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.100.2@KRBTEST.COM for kudu/127.31.100.62@KRBTEST.COM
I20260812 05:20:31.825744 32144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0waFvT/build/release/bin/kudu as pid 4107
I20260812 05:20:31.825824 32144 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786512017613366-32144-0/minicluster-data/ts-1/wal/instance
I20260812 05:20:31.826005 32144 external_mini_cluster.cc:1428] Setting key e24aafad6832c1926a4f01dc2f3af540
I20260812 05:20:31.827287 3827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:31.819680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.100.2:59905 (local address 127.31.100.62:33135)
0812 05:20:31.819758 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0812 05:20:31.819760 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 05:20:31.819891 (+ 131us) server_negotiation.cc:407] Connection header received
0812 05:20:31.820320 (+ 429us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:20:31.820322 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:20:31.820332 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:20:31.820379 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:20:31.821231 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:31.821806 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:31.822727 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:31.822913 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:31.822958 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:31.824914 (+ 1956us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 05:20:31.824915 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 05:20:31.824915 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 05:20:31.824922 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 05:20:31.826019 (+ 1097us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:20:31.826457 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:20:31.826458 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:20:31.826458 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:20:31.826508 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:20:31.826736 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:20:31.826737 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:20:31.826737 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:20:31.826838 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 05:20:31.826893 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0812 05:20:31.827165 (+ 272us) server_negotiation.cc:299] Negotiation successful
0812 05:20:31.827227 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260812 05:20:31.827515 4325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:31.819639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.62:33135 (local address 127.31.100.2:59905)
0812 05:20:31.819790 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0812 05:20:31.819823 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0812 05:20:31.820246 (+ 423us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:20:31.820453 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:20:31.820456 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:20:31.820967 (+ 511us) client_negotiation.cc:189] Negotiated authn=SASL
0812 05:20:31.821152 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:20:31.821156 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:20:31.821888 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:20:31.821889 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:20:31.822662 (+ 773us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:20:31.822664 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:20:31.822995 (+ 331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:20:31.822996 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:20:31.823118 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:31.823549 (+ 431us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 05:20:31.823557 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0812 05:20:31.824828 (+ 1271us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 05:20:31.826111 (+ 1283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 05:20:31.826118 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 05:20:31.826120 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 05:20:31.826388 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 05:20:31.826566 (+ 178us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 05:20:31.826567 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 05:20:31.826567 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 05:20:31.826681 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 05:20:31.826896 (+ 215us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 05:20:31.826902 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 05:20:31.827045 (+ 143us) client_negotiation.cc:769] Sending connection context
0812 05:20:31.827148 (+ 103us) client_negotiation.cc:240] Negotiation successful
0812 05:20:31.827300 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":73,"threads_started":1}
I20260812 05:20:31.828716 4320 heartbeater.cc:347] Connected to a master server at 127.31.100.62:33135
I20260812 05:20:31.828816 4320 heartbeater.cc:464] Registering TS with master...
I20260812 05:20:31.828990 4320 heartbeater.cc:511] Master 127.31.100.62:33135 requested a full tablet report, sending...
I20260812 05:20:31.829308 3758 ts_manager.cc:194] Registered new tserver with Master: 972d0a1fb5a147af83b2130f29f6df58 (127.31.100.2:44687)
I20260812 05:20:31.829698 3758 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.100.2@KRBTEST.COM'} at 127.31.100.2:59905
WARNING: no policy specified for kudu/127.31.100.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.100.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.100.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.3.keytab.
Entry for principal kudu/127.31.100.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.3.keytab.
I20260812 05:20:31.849320 32144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0waFvT/build/release/bin/kudu
/tmp/dist-test-task0waFvT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786512017613366-32144-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786512017613366-32144-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786512017613366-32144-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786512017613366-32144-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.100.3:0
--local_ip_for_outbound_sockets=127.31.100.3
--webserver_interface=127.31.100.3
--webserver_port=0
--tserver_master_addrs=127.31.100.62:33135
--keytab_file=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.3.keytab
--principal=kudu/127.31.100.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.100.20:36977
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786512017613366-32144-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786512017613366-32144-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786512017613366-32144-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 05:20:31.927668 4329 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 05:20:31.927886 4329 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 05:20:31.927910 4329 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 05:20:31.929060 4329 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 05:20:31.929102 4329 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 05:20:31.929126 4329 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.100.3
I20260812 05:20:31.930543 4329 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.100.20:36977
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786512017613366-32144-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786512017613366-32144-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.3.keytab
--principal=kudu/127.31.100.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.100.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786512017613366-32144-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.100.3
--webserver_port=0
--tserver_master_addrs=127.31.100.62:33135
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4329
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.100.3
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786512017613366-32144-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 04:20:34 UTC on ccycloud.quasar-phqdqp.root.comops.site
build id 4581871
I20260812 05:20:31.930745 4329 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 05:20:31.930949 4329 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 05:20:31.935629 4340 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 05:20:31.935657 4343 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 05:20:31.935638 4346 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 05:20:31.936038 4329 server_base.cc:1034] running on GCE node
I20260812 05:20:31.936221 4329 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 05:20:31.936457 4329 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 05:20:31.937595 4329 hybrid_clock.cc:648] HybridClock initialized: now 1786512031937576 us; error 32 us; skew 500 ppm
Aug 12 05:20:31 dist-test-slave-42z9 krb5kdc[3689](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512031, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.100.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 05:20:31.939850 4329 init.cc:377] Logged in from keytab as kudu/127.31.100.3@KRBTEST.COM (short username kudu)
I20260812 05:20:31.940268 4329 webserver.cc:460] Webserver started at http://127.31.100.3:40587/ using document root <none> and password file <none>
I20260812 05:20:31.940486 4329 fs_manager.cc:359] Metadata directory not provided
I20260812 05:20:31.940533 4329 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 05:20:31.940652 4329 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 05:20:31.941636 4329 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786512017613366-32144-0/minicluster-data/ts-2/data/instance:
uuid: "defc5645e9fe4f9b8490993a2b058504"
format_stamp: "Formatted at 2026-08-12 05:20:31 on dist-test-slave-42z9"
server_key: "8056ef520c9f1c78ea2defdd1b8062c5"
server_key_iv: "a80981a40ed14be00e88b46bcca8105f"
server_key_version: "encryptionkey@0"
I20260812 05:20:31.941972 4329 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786512017613366-32144-0/minicluster-data/ts-2/wal/instance:
uuid: "defc5645e9fe4f9b8490993a2b058504"
format_stamp: "Formatted at 2026-08-12 05:20:31 on dist-test-slave-42z9"
server_key: "8056ef520c9f1c78ea2defdd1b8062c5"
server_key_iv: "a80981a40ed14be00e88b46bcca8105f"
server_key_version: "encryptionkey@0"
I20260812 05:20:31.943184 4329 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260812 05:20:31.943892 4356 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 05:20:31.944054 4329 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260812 05:20:31.944142 4329 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786512017613366-32144-0/minicluster-data/ts-2/data,/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786512017613366-32144-0/minicluster-data/ts-2/wal
uuid: "defc5645e9fe4f9b8490993a2b058504"
format_stamp: "Formatted at 2026-08-12 05:20:31 on dist-test-slave-42z9"
server_key: "8056ef520c9f1c78ea2defdd1b8062c5"
server_key_iv: "a80981a40ed14be00e88b46bcca8105f"
server_key_version: "encryptionkey@0"
I20260812 05:20:31.944195 4329 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786512017613366-32144-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786512017613366-32144-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786512017613366-32144-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 05:20:31.956000 4329 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 05:20:31.957595 4329 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 05:20:31.957700 4329 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 05:20:31.957873 4329 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 05:20:31.958108 4329 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 05:20:31.958137 4329 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 05:20:31.958155 4329 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 05:20:31.958171 4329 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 05:20:31.963707 4329 rpc_server.cc:307] RPC server started. Bound to: 127.31.100.3:34631
I20260812 05:20:31.963761 4538 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.100.3:34631 every 8 connection(s)
I20260812 05:20:31.964036 4329 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786512017613366-32144-0/minicluster-data/ts-2/data/info.pb
Aug 12 05:20:31 dist-test-slave-42z9 krb5kdc[3689](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512031, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.100.3@KRBTEST.COM for kudu/127.31.100.62@KRBTEST.COM
I20260812 05:20:31.972435 3827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:31.964619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.100.3:49431 (local address 127.31.100.62:33135)
0812 05:20:31.964678 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0812 05:20:31.964680 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 05:20:31.964830 (+ 150us) server_negotiation.cc:407] Connection header received
0812 05:20:31.965296 (+ 466us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:20:31.965297 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:20:31.965308 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:20:31.965361 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:20:31.966295 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:31.966860 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:31.967875 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:31.968115 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:31.968173 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:31.970185 (+ 2012us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 05:20:31.970186 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 05:20:31.970187 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 05:20:31.970194 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 05:20:31.971403 (+ 1209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:20:31.971759 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:20:31.971760 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:20:31.971760 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:20:31.971804 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:20:31.971995 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:20:31.971995 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:20:31.971996 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:20:31.972086 (+ 90us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 05:20:31.972128 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0812 05:20:31.972320 (+ 192us) server_negotiation.cc:299] Negotiation successful
0812 05:20:31.972374 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260812 05:20:31.972642 4545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:31.964583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.62:33135 (local address 127.31.100.3:49431)
0812 05:20:31.964731 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0812 05:20:31.964769 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0812 05:20:31.965198 (+ 429us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:20:31.965412 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:20:31.965415 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:20:31.966049 (+ 634us) client_negotiation.cc:189] Negotiated authn=SASL
0812 05:20:31.966225 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:20:31.966230 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:20:31.966951 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:20:31.966952 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:20:31.967738 (+ 786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:20:31.967760 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:20:31.968180 (+ 420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:20:31.968181 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:20:31.968275 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:31.968742 (+ 467us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 05:20:31.968751 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0812 05:20:31.970091 (+ 1340us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 05:20:31.971486 (+ 1395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 05:20:31.971493 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 05:20:31.971495 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 05:20:31.971698 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 05:20:31.971849 (+ 151us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 05:20:31.971850 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 05:20:31.971850 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 05:20:31.971952 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 05:20:31.972126 (+ 174us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 05:20:31.972129 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 05:20:31.972229 (+ 100us) client_negotiation.cc:769] Sending connection context
0812 05:20:31.972302 (+ 73us) client_negotiation.cc:240] Negotiation successful
0812 05:20:31.972412 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":80,"threads_started":1}
I20260812 05:20:31.973073 4540 heartbeater.cc:347] Connected to a master server at 127.31.100.62:33135
I20260812 05:20:31.973157 4540 heartbeater.cc:464] Registering TS with master...
I20260812 05:20:31.973297 4540 heartbeater.cc:511] Master 127.31.100.62:33135 requested a full tablet report, sending...
I20260812 05:20:31.973618 32144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0waFvT/build/release/bin/kudu as pid 4329
I20260812 05:20:31.973662 3758 ts_manager.cc:194] Registered new tserver with Master: defc5645e9fe4f9b8490993a2b058504 (127.31.100.3:34631)
I20260812 05:20:31.973695 32144 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786512017613366-32144-0/minicluster-data/ts-2/wal/instance
I20260812 05:20:31.973850 32144 external_mini_cluster.cc:1428] Setting key aa7cc57826b53652c007c5f731aa48ef
I20260812 05:20:31.974105 3758 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.100.3@KRBTEST.COM'} at 127.31.100.3:49431
I20260812 05:20:31.974697 32144 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260812 05:20:31.986990 3827 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 05:20:31.986197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33216 (local address 127.31.100.62:33135)
0812 05:20:31.986259 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0812 05:20:31.986261 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 05:20:31.986341 (+ 80us) server_negotiation.cc:407] Connection header received
0812 05:20:31.986390 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:20:31.986391 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:20:31.986399 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:20:31.986443 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:20:31.986932 (+ 489us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33216: BlockingRecv error: recv got EOF from 127.0.0.1:33216 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
I20260812 05:20:31.987789 32144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0waFvT/build/release/bin/kudu with pid 3863
I20260812 05:20:32.004091 32144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0waFvT/build/release/bin/kudu with pid 4107
I20260812 05:20:32.013052 32144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0waFvT/build/release/bin/kudu with pid 4329
I20260812 05:20:32.021258 32144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0waFvT/build/release/bin/kudu with pid 3710
2026-08-12T05:20:32Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (1956 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786512017613366-32144-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 05:20:32 dist-test-slave-42z9 krb5kdc[4557](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 05:20:32 dist-test-slave-42z9 krb5kdc[4557](info): set up 2 sockets
Aug 12 05:20:32 dist-test-slave-42z9 krb5kdc[4557](info): commencing operation
krb5kdc: starting...
W20260812 05:20:33.196431 32144 mini_kdc.cc:121] Time spent starting KDC: real 1.153s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786512017613366-32144-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786512017613366-32144-0/krb5kdc/test-user.keytab.
Aug 12 05:20:33 dist-test-slave-42z9 krb5kdc[4557](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512033, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T05:20:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T05:20:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.100.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.100.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.100.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.62.keytab.
Entry for principal kudu/127.31.100.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.62.keytab.
I20260812 05:20:33.307772 32144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0waFvT/build/release/bin/kudu
/tmp/dist-test-task0waFvT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786512017613366-32144-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786512017613366-32144-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786512017613366-32144-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786512017613366-32144-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.100.62:44201
--webserver_interface=127.31.100.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.62.keytab
--principal=kudu/127.31.100.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.100.20:37923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.100.62:44201
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786512017613366-32144-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786512017613366-32144-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786512017613366-32144-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 05:20:33.387228 4573 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 05:20:33.387421 4573 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 05:20:33.387442 4573 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 05:20:33.388563 4573 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 05:20:33.388604 4573 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 05:20:33.388617 4573 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 05:20:33.388629 4573 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 05:20:33.388653 4573 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 05:20:33.390136 4573 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.100.20:37923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786512017613366-32144-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786512017613366-32144-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.100.62:44201
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.62.keytab
--principal=kudu/127.31.100.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.100.62:44201
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786512017613366-32144-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.100.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4573
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786512017613366-32144-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 04:20:34 UTC on ccycloud.quasar-phqdqp.root.comops.site
build id 4581871
I20260812 05:20:33.390357 4573 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 05:20:33.390560 4573 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 05:20:33.395159 4579 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 05:20:33.395124 4584 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 05:20:33.395237 4573 server_base.cc:1034] running on GCE node
W20260812 05:20:33.395133 4581 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 05:20:33.395572 4573 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 05:20:33.395864 4573 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 05:20:33.397011 4573 hybrid_clock.cc:648] HybridClock initialized: now 1786512033396994 us; error 35 us; skew 500 ppm
Aug 12 05:20:33 dist-test-slave-42z9 krb5kdc[4557](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512033, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.100.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 05:20:33.399387 4573 init.cc:377] Logged in from keytab as kudu/127.31.100.62@KRBTEST.COM (short username kudu)
I20260812 05:20:33.399824 4573 webserver.cc:460] Webserver started at http://127.31.100.62:35663/ using document root <none> and password file <none>
I20260812 05:20:33.399995 4573 fs_manager.cc:359] Metadata directory not provided
I20260812 05:20:33.400033 4573 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 05:20:33.400117 4573 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 05:20:33.401034 4573 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786512017613366-32144-0/minicluster-data/master-0/data/instance:
uuid: "6c0647e14dd344c4b462c351bc73dae3"
format_stamp: "Formatted at 2026-08-12 05:20:33 on dist-test-slave-42z9"
server_key: "91fccdc73cc9e0ae74c047ce2d48f8fe"
server_key_iv: "d6b224f5cdcbb6da9d285437337f57b8"
server_key_version: "encryptionkey@0"
I20260812 05:20:33.401314 4573 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786512017613366-32144-0/minicluster-data/master-0/wal/instance:
uuid: "6c0647e14dd344c4b462c351bc73dae3"
format_stamp: "Formatted at 2026-08-12 05:20:33 on dist-test-slave-42z9"
server_key: "91fccdc73cc9e0ae74c047ce2d48f8fe"
server_key_iv: "d6b224f5cdcbb6da9d285437337f57b8"
server_key_version: "encryptionkey@0"
I20260812 05:20:33.402549 4573 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260812 05:20:33.403196 4598 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 05:20:33.403373 4573 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260812 05:20:33.403487 4573 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786512017613366-32144-0/minicluster-data/master-0/data,/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786512017613366-32144-0/minicluster-data/master-0/wal
uuid: "6c0647e14dd344c4b462c351bc73dae3"
format_stamp: "Formatted at 2026-08-12 05:20:33 on dist-test-slave-42z9"
server_key: "91fccdc73cc9e0ae74c047ce2d48f8fe"
server_key_iv: "d6b224f5cdcbb6da9d285437337f57b8"
server_key_version: "encryptionkey@0"
I20260812 05:20:33.403539 4573 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786512017613366-32144-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786512017613366-32144-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786512017613366-32144-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 05:20:33.429955 4573 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 05:20:33.431785 4573 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 05:20:33.431890 4573 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 05:20:33.435516 4573 rpc_server.cc:307] RPC server started. Bound to: 127.31.100.62:44201
I20260812 05:20:33.435554 4686 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.100.62:44201 every 8 connection(s)
I20260812 05:20:33.436038 4573 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786512017613366-32144-0/minicluster-data/master-0/data/info.pb
I20260812 05:20:33.436671 4689 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 05:20:33.438995 4689 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6c0647e14dd344c4b462c351bc73dae3: Bootstrap starting.
I20260812 05:20:33.439589 4689 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6c0647e14dd344c4b462c351bc73dae3: Neither blocks nor log segments found. Creating new log.
I20260812 05:20:33.439862 4689 log.cc:824] T 00000000000000000000000000000000 P 6c0647e14dd344c4b462c351bc73dae3: Log is configured to *not* fsync() on all Append() calls
I20260812 05:20:33.440493 4689 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6c0647e14dd344c4b462c351bc73dae3: No bootstrap required, opened a new log
I20260812 05:20:33.442113 4689 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6c0647e14dd344c4b462c351bc73dae3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c0647e14dd344c4b462c351bc73dae3" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 44201 } }
I20260812 05:20:33.442247 4689 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6c0647e14dd344c4b462c351bc73dae3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 05:20:33.442269 4689 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6c0647e14dd344c4b462c351bc73dae3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c0647e14dd344c4b462c351bc73dae3, State: Initialized, Role: FOLLOWER
I20260812 05:20:33.442408 4689 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6c0647e14dd344c4b462c351bc73dae3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c0647e14dd344c4b462c351bc73dae3" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 44201 } }
I20260812 05:20:33.442471 4689 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6c0647e14dd344c4b462c351bc73dae3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 05:20:33.442492 4689 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6c0647e14dd344c4b462c351bc73dae3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 05:20:33.442525 4689 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6c0647e14dd344c4b462c351bc73dae3 [term 0 FOLLOWER]: Advancing to term 1
I20260812 05:20:33.442752 32144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0waFvT/build/release/bin/kudu as pid 4573
I20260812 05:20:33.442847 32144 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786512017613366-32144-0/minicluster-data/master-0/wal/instance
I20260812 05:20:33.442996 32144 external_mini_cluster.cc:1428] Setting key bbd6e7ed16e3ca845eea6de40762d2d4
I20260812 05:20:33.443493 4689 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6c0647e14dd344c4b462c351bc73dae3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c0647e14dd344c4b462c351bc73dae3" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 44201 } }
I20260812 05:20:33.443606 4689 leader_election.cc:302] T 00000000000000000000000000000000 P 6c0647e14dd344c4b462c351bc73dae3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6c0647e14dd344c4b462c351bc73dae3; no voters:
I20260812 05:20:33.443761 4689 leader_election.cc:288] T 00000000000000000000000000000000 P 6c0647e14dd344c4b462c351bc73dae3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 05:20:33.443799 4697 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6c0647e14dd344c4b462c351bc73dae3 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 05:20:33.443953 4697 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6c0647e14dd344c4b462c351bc73dae3 [term 1 LEADER]: Becoming Leader. State: Replica: 6c0647e14dd344c4b462c351bc73dae3, State: Running, Role: LEADER
I20260812 05:20:33.444031 4689 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6c0647e14dd344c4b462c351bc73dae3 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 05:20:33.444141 4697 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6c0647e14dd344c4b462c351bc73dae3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c0647e14dd344c4b462c351bc73dae3" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 44201 } }
I20260812 05:20:33.444567 4699 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c0647e14dd344c4b462c351bc73dae3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6c0647e14dd344c4b462c351bc73dae3. Latest consensus state: current_term: 1 leader_uuid: "6c0647e14dd344c4b462c351bc73dae3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c0647e14dd344c4b462c351bc73dae3" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 44201 } } }
I20260812 05:20:33.444648 4699 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c0647e14dd344c4b462c351bc73dae3 [sys.catalog]: This master's current role is: LEADER
I20260812 05:20:33.444782 4698 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c0647e14dd344c4b462c351bc73dae3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6c0647e14dd344c4b462c351bc73dae3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c0647e14dd344c4b462c351bc73dae3" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 44201 } } }
I20260812 05:20:33.444826 4698 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c0647e14dd344c4b462c351bc73dae3 [sys.catalog]: This master's current role is: LEADER
I20260812 05:20:33.445148 4710 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 05:20:33.445672 4710 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 05:20:33.447160 4710 catalog_manager.cc:1347] Generated new cluster ID: 1acde7950b5a41a28cf12006219962ff
I20260812 05:20:33.447208 4710 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 12 05:20:33 dist-test-slave-42z9 krb5kdc[4557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512033, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.100.62@KRBTEST.COM
I20260812 05:20:33.451953 4710 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 05:20:33.452399 4710 catalog_manager.cc:1504] Loading token signing keys...
I20260812 05:20:33.454246 4696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:33.443421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59310 (local address 127.31.100.62:44201)
0812 05:20:33.443602 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0812 05:20:33.443612 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 05:20:33.443644 (+ 32us) server_negotiation.cc:407] Connection header received
0812 05:20:33.443755 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:20:33.443760 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:20:33.443812 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:20:33.444013 (+ 201us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:20:33.444761 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:33.446145 (+ 1384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:33.447143 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:33.447699 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:33.447805 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:33.450540 (+ 2735us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 05:20:33.450544 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 05:20:33.450544 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 05:20:33.450557 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0812 05:20:33.452448 (+ 1891us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:20:33.452821 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:20:33.452825 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:20:33.452825 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:20:33.452949 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:20:33.453125 (+ 176us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:20:33.453126 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:20:33.453126 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:20:33.453394 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 05:20:33.453628 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0812 05:20:33.453700 (+ 72us) server_negotiation.cc:299] Negotiation successful
0812 05:20:33.454011 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":81,"threads_started":1}
I20260812 05:20:33.455273 32144 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 05:20:33.460285 4710 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6c0647e14dd344c4b462c351bc73dae3: Generated new TSK 0
I20260812 05:20:33.460457 4710 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 05:20:33.473858 4696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:33.470094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59314 (local address 127.31.100.62:44201)
0812 05:20:33.470149 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0812 05:20:33.470152 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 05:20:33.470240 (+ 88us) server_negotiation.cc:407] Connection header received
0812 05:20:33.470283 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:20:33.470284 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:20:33.470296 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:20:33.470366 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:20:33.470700 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:33.471548 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:33.472293 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:33.472626 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:33.472687 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:33.473299 (+ 612us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 05:20:33.473300 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 05:20:33.473300 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 05:20:33.473589 (+ 289us) server_negotiation.cc:922] calling sasl_server_start()
0812 05:20:33.473609 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 05:20:33.473618 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 05:20:33.473676 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0812 05:20:33.473752 (+ 76us) server_negotiation.cc:299] Negotiation successful
0812 05:20:33.473779 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260812 05:20:33.476089 32144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0waFvT/build/release/bin/kudu with pid 4573
2026-08-12T05:20:33Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1460 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786512017613366-32144-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 05:20:33 dist-test-slave-42z9 krb5kdc[4739](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 05:20:33 dist-test-slave-42z9 krb5kdc[4739](info): set up 2 sockets
Aug 12 05:20:33 dist-test-slave-42z9 krb5kdc[4739](info): commencing operation
krb5kdc: starting...
W20260812 05:20:34.642606 32144 mini_kdc.cc:121] Time spent starting KDC: real 1.139s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786512017613366-32144-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786512017613366-32144-0/krb5kdc/test-user.keytab.
Aug 12 05:20:34 dist-test-slave-42z9 krb5kdc[4739](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512034, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T05:20:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T05:20:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.100.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.100.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.100.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.62.keytab.
Entry for principal kudu/127.31.100.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.62.keytab.
I20260812 05:20:34.755852 32144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0waFvT/build/release/bin/kudu
/tmp/dist-test-task0waFvT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786512017613366-32144-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786512017613366-32144-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786512017613366-32144-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786512017613366-32144-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.100.62:42235
--webserver_interface=127.31.100.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.62.keytab
--principal=kudu/127.31.100.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.100.20:38789
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.100.62:42235
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786512017613366-32144-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786512017613366-32144-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786512017613366-32144-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 05:20:34.835644 4759 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 05:20:34.835825 4759 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 05:20:34.835844 4759 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 05:20:34.836947 4759 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 05:20:34.836987 4759 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 05:20:34.837000 4759 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 05:20:34.837013 4759 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 05:20:34.837036 4759 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 05:20:34.838663 4759 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.100.20:38789
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786512017613366-32144-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786512017613366-32144-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.100.62:42235
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.62.keytab
--principal=kudu/127.31.100.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.100.62:42235
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786512017613366-32144-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.100.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4759
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786512017613366-32144-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 04:20:34 UTC on ccycloud.quasar-phqdqp.root.comops.site
build id 4581871
I20260812 05:20:34.838903 4759 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 05:20:34.839128 4759 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 05:20:34.843778 4766 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 05:20:34.843778 4769 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 05:20:34.843797 4765 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 05:20:34.843967 4759 server_base.cc:1034] running on GCE node
I20260812 05:20:34.844246 4759 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 05:20:34.844529 4759 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 05:20:34.845659 4759 hybrid_clock.cc:648] HybridClock initialized: now 1786512034845648 us; error 27 us; skew 500 ppm
Aug 12 05:20:34 dist-test-slave-42z9 krb5kdc[4739](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512034, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.100.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 05:20:34.851483 4759 init.cc:377] Logged in from keytab as kudu/127.31.100.62@KRBTEST.COM (short username kudu)
I20260812 05:20:34.851909 4759 webserver.cc:460] Webserver started at http://127.31.100.62:36389/ using document root <none> and password file <none>
I20260812 05:20:34.852108 4759 fs_manager.cc:359] Metadata directory not provided
I20260812 05:20:34.852154 4759 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 05:20:34.852277 4759 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 05:20:34.853278 4759 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786512017613366-32144-0/minicluster-data/master-0/data/instance:
uuid: "0dd9dcd9102541b581c00476c4a3a106"
format_stamp: "Formatted at 2026-08-12 05:20:34 on dist-test-slave-42z9"
server_key: "9ee813abba3c7ec2e417f4df342d8f97"
server_key_iv: "bee73ee03688e36fac7df50befe049ed"
server_key_version: "encryptionkey@0"
I20260812 05:20:34.853631 4759 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786512017613366-32144-0/minicluster-data/master-0/wal/instance:
uuid: "0dd9dcd9102541b581c00476c4a3a106"
format_stamp: "Formatted at 2026-08-12 05:20:34 on dist-test-slave-42z9"
server_key: "9ee813abba3c7ec2e417f4df342d8f97"
server_key_iv: "bee73ee03688e36fac7df50befe049ed"
server_key_version: "encryptionkey@0"
I20260812 05:20:34.854867 4759 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260812 05:20:34.855612 4780 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 05:20:34.855751 4759 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260812 05:20:34.855834 4759 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786512017613366-32144-0/minicluster-data/master-0/data,/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786512017613366-32144-0/minicluster-data/master-0/wal
uuid: "0dd9dcd9102541b581c00476c4a3a106"
format_stamp: "Formatted at 2026-08-12 05:20:34 on dist-test-slave-42z9"
server_key: "9ee813abba3c7ec2e417f4df342d8f97"
server_key_iv: "bee73ee03688e36fac7df50befe049ed"
server_key_version: "encryptionkey@0"
I20260812 05:20:34.855894 4759 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786512017613366-32144-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786512017613366-32144-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786512017613366-32144-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 05:20:34.875404 4759 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 05:20:34.877146 4759 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 05:20:34.877276 4759 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 05:20:34.881232 4759 rpc_server.cc:307] RPC server started. Bound to: 127.31.100.62:42235
I20260812 05:20:34.881281 4868 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.100.62:42235 every 8 connection(s)
I20260812 05:20:34.881624 4759 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786512017613366-32144-0/minicluster-data/master-0/data/info.pb
I20260812 05:20:34.882582 4869 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 05:20:34.884833 4869 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0dd9dcd9102541b581c00476c4a3a106: Bootstrap starting.
I20260812 05:20:34.885375 4869 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0dd9dcd9102541b581c00476c4a3a106: Neither blocks nor log segments found. Creating new log.
I20260812 05:20:34.885676 4869 log.cc:824] T 00000000000000000000000000000000 P 0dd9dcd9102541b581c00476c4a3a106: Log is configured to *not* fsync() on all Append() calls
I20260812 05:20:34.886319 4869 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0dd9dcd9102541b581c00476c4a3a106: No bootstrap required, opened a new log
I20260812 05:20:34.887854 4869 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0dd9dcd9102541b581c00476c4a3a106 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0dd9dcd9102541b581c00476c4a3a106" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 42235 } }
I20260812 05:20:34.887997 4869 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0dd9dcd9102541b581c00476c4a3a106 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 05:20:34.888018 4869 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0dd9dcd9102541b581c00476c4a3a106 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0dd9dcd9102541b581c00476c4a3a106, State: Initialized, Role: FOLLOWER
I20260812 05:20:34.888120 4869 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0dd9dcd9102541b581c00476c4a3a106 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0dd9dcd9102541b581c00476c4a3a106" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 42235 } }
I20260812 05:20:34.888170 4869 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0dd9dcd9102541b581c00476c4a3a106 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 05:20:34.888188 4869 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0dd9dcd9102541b581c00476c4a3a106 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 05:20:34.888217 4869 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0dd9dcd9102541b581c00476c4a3a106 [term 0 FOLLOWER]: Advancing to term 1
I20260812 05:20:34.888945 4869 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0dd9dcd9102541b581c00476c4a3a106 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0dd9dcd9102541b581c00476c4a3a106" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 42235 } }
I20260812 05:20:34.889032 4869 leader_election.cc:302] T 00000000000000000000000000000000 P 0dd9dcd9102541b581c00476c4a3a106 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0dd9dcd9102541b581c00476c4a3a106; no voters:
I20260812 05:20:34.889146 4869 leader_election.cc:288] T 00000000000000000000000000000000 P 0dd9dcd9102541b581c00476c4a3a106 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 05:20:34.889194 4874 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0dd9dcd9102541b581c00476c4a3a106 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 05:20:34.889353 4874 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0dd9dcd9102541b581c00476c4a3a106 [term 1 LEADER]: Becoming Leader. State: Replica: 0dd9dcd9102541b581c00476c4a3a106, State: Running, Role: LEADER
I20260812 05:20:34.889364 4869 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0dd9dcd9102541b581c00476c4a3a106 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 05:20:34.889482 4874 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0dd9dcd9102541b581c00476c4a3a106 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0dd9dcd9102541b581c00476c4a3a106" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 42235 } }
I20260812 05:20:34.889878 4877 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0dd9dcd9102541b581c00476c4a3a106 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0dd9dcd9102541b581c00476c4a3a106. Latest consensus state: current_term: 1 leader_uuid: "0dd9dcd9102541b581c00476c4a3a106" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0dd9dcd9102541b581c00476c4a3a106" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 42235 } } }
I20260812 05:20:34.889897 4875 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0dd9dcd9102541b581c00476c4a3a106 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0dd9dcd9102541b581c00476c4a3a106" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0dd9dcd9102541b581c00476c4a3a106" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 42235 } } }
I20260812 05:20:34.889962 4877 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0dd9dcd9102541b581c00476c4a3a106 [sys.catalog]: This master's current role is: LEADER
I20260812 05:20:34.889979 4875 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0dd9dcd9102541b581c00476c4a3a106 [sys.catalog]: This master's current role is: LEADER
I20260812 05:20:34.890388 4884 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 05:20:34.890489 32144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0waFvT/build/release/bin/kudu as pid 4759
I20260812 05:20:34.890594 32144 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786512017613366-32144-0/minicluster-data/master-0/wal/instance
I20260812 05:20:34.890767 32144 external_mini_cluster.cc:1428] Setting key b4c23981901654e8ce3ddef51e07a5bd
I20260812 05:20:34.890888 4884 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 05:20:34.893034 4884 catalog_manager.cc:1347] Generated new cluster ID: 9fd20a9110b94af1aaf1d83bf80f05f6
I20260812 05:20:34.893086 4884 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 12 05:20:34 dist-test-slave-42z9 krb5kdc[4739](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512034, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.100.62@KRBTEST.COM
I20260812 05:20:34.902151 4906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:34.891324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34308 (local address 127.31.100.62:42235)
0812 05:20:34.891505 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0812 05:20:34.891514 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 05:20:34.891659 (+ 145us) server_negotiation.cc:407] Connection header received
0812 05:20:34.891777 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:20:34.891784 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:20:34.891837 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:20:34.891975 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:20:34.892687 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:34.894093 (+ 1406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:34.895182 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:34.895827 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:34.895938 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:34.898686 (+ 2748us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 05:20:34.898690 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 05:20:34.898691 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 05:20:34.898702 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0812 05:20:34.900573 (+ 1871us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:20:34.900947 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:20:34.900950 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:20:34.900950 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:20:34.901049 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:20:34.901235 (+ 186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:20:34.901236 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:20:34.901236 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:20:34.901553 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 05:20:34.901690 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0812 05:20:34.901803 (+ 113us) server_negotiation.cc:299] Negotiation successful
0812 05:20:34.901931 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":71,"threads_started":1}
I20260812 05:20:34.908735 4884 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 05:20:34.909242 4884 catalog_manager.cc:1504] Loading token signing keys...
I20260812 05:20:34.912678 4884 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0dd9dcd9102541b581c00476c4a3a106: Generated new TSK 0
I20260812 05:20:34.912863 4884 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.31.100.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.100.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.100.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.1.keytab.
Entry for principal kudu/127.31.100.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.1.keytab.
I20260812 05:20:34.946095 32144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0waFvT/build/release/bin/kudu
/tmp/dist-test-task0waFvT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786512017613366-32144-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786512017613366-32144-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786512017613366-32144-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786512017613366-32144-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.100.1:0
--local_ip_for_outbound_sockets=127.31.100.1
--webserver_interface=127.31.100.1
--webserver_port=0
--tserver_master_addrs=127.31.100.62:42235
--keytab_file=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.1.keytab
--principal=kudu/127.31.100.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.100.20:38789
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786512017613366-32144-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786512017613366-32144-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786512017613366-32144-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 05:20:35.052824 4912 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 05:20:35.053009 4912 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 05:20:35.053031 4912 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 05:20:35.054157 4912 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 05:20:35.054196 4912 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 05:20:35.054220 4912 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.100.1
I20260812 05:20:35.055645 4912 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.100.20:38789
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786512017613366-32144-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786512017613366-32144-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.1.keytab
--principal=kudu/127.31.100.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.100.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786512017613366-32144-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.100.1
--webserver_port=0
--tserver_master_addrs=127.31.100.62:42235
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4912
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.100.1
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786512017613366-32144-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 04:20:34 UTC on ccycloud.quasar-phqdqp.root.comops.site
build id 4581871
I20260812 05:20:35.055859 4912 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 05:20:35.056068 4912 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 05:20:35.060813 4926 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 05:20:35.060819 4924 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 05:20:35.060979 4912 server_base.cc:1034] running on GCE node
W20260812 05:20:35.060822 4921 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 05:20:35.061210 4912 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 05:20:35.061407 4912 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 05:20:35.062572 4912 hybrid_clock.cc:648] HybridClock initialized: now 1786512035062559 us; error 28 us; skew 500 ppm
Aug 12 05:20:35 dist-test-slave-42z9 krb5kdc[4739](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512035, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.100.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 05:20:35.064893 4912 init.cc:377] Logged in from keytab as kudu/127.31.100.1@KRBTEST.COM (short username kudu)
I20260812 05:20:35.065299 4912 webserver.cc:460] Webserver started at http://127.31.100.1:35193/ using document root <none> and password file <none>
I20260812 05:20:35.065553 4912 fs_manager.cc:359] Metadata directory not provided
I20260812 05:20:35.065605 4912 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 05:20:35.065729 4912 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 05:20:35.066723 4912 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786512017613366-32144-0/minicluster-data/ts-0/data/instance:
uuid: "43863ed5260a43f884343e8c47e77e42"
format_stamp: "Formatted at 2026-08-12 05:20:35 on dist-test-slave-42z9"
server_key: "7252272a5eb69d87309132b4393d8274"
server_key_iv: "237aa18dfc7f81815b9dd75af52126d1"
server_key_version: "encryptionkey@0"
I20260812 05:20:35.067041 4912 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786512017613366-32144-0/minicluster-data/ts-0/wal/instance:
uuid: "43863ed5260a43f884343e8c47e77e42"
format_stamp: "Formatted at 2026-08-12 05:20:35 on dist-test-slave-42z9"
server_key: "7252272a5eb69d87309132b4393d8274"
server_key_iv: "237aa18dfc7f81815b9dd75af52126d1"
server_key_version: "encryptionkey@0"
I20260812 05:20:35.068349 4912 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260812 05:20:35.068988 4938 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 05:20:35.069123 4912 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260812 05:20:35.069206 4912 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786512017613366-32144-0/minicluster-data/ts-0/data,/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786512017613366-32144-0/minicluster-data/ts-0/wal
uuid: "43863ed5260a43f884343e8c47e77e42"
format_stamp: "Formatted at 2026-08-12 05:20:35 on dist-test-slave-42z9"
server_key: "7252272a5eb69d87309132b4393d8274"
server_key_iv: "237aa18dfc7f81815b9dd75af52126d1"
server_key_version: "encryptionkey@0"
I20260812 05:20:35.069265 4912 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786512017613366-32144-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786512017613366-32144-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786512017613366-32144-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 05:20:35.095279 4912 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 05:20:35.097114 4912 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 05:20:35.097255 4912 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 05:20:35.097492 4912 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 05:20:35.097802 4912 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 05:20:35.097836 4912 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 05:20:35.097865 4912 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 05:20:35.097883 4912 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 05:20:35.103507 4912 rpc_server.cc:307] RPC server started. Bound to: 127.31.100.1:35107
I20260812 05:20:35.103574 5125 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.100.1:35107 every 8 connection(s)
I20260812 05:20:35.103849 4912 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786512017613366-32144-0/minicluster-data/ts-0/data/info.pb
Aug 12 05:20:35 dist-test-slave-42z9 krb5kdc[4739](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512035, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.100.1@KRBTEST.COM for kudu/127.31.100.62@KRBTEST.COM
I20260812 05:20:35.111778 32144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0waFvT/build/release/bin/kudu as pid 4912
I20260812 05:20:35.111876 32144 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786512017613366-32144-0/minicluster-data/ts-0/wal/instance
I20260812 05:20:35.112064 32144 external_mini_cluster.cc:1428] Setting key 58780d00749cb7ad1abb189e1317a85e
I20260812 05:20:35.112393 4906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:35.104512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.100.1:49843 (local address 127.31.100.62:42235)
0812 05:20:35.104574 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0812 05:20:35.104576 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 05:20:35.104759 (+ 183us) server_negotiation.cc:407] Connection header received
0812 05:20:35.105217 (+ 458us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:20:35.105218 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:20:35.105231 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:20:35.105279 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:20:35.106158 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:35.106655 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:35.107596 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:35.107796 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:35.107845 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:35.109815 (+ 1970us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 05:20:35.109817 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 05:20:35.109817 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 05:20:35.109823 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0812 05:20:35.111061 (+ 1238us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:20:35.111381 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:20:35.111382 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:20:35.111382 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:20:35.111426 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:20:35.111673 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:20:35.111673 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:20:35.111674 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:20:35.111804 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 05:20:35.111872 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0812 05:20:35.112202 (+ 330us) server_negotiation.cc:299] Negotiation successful
0812 05:20:35.112298 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260812 05:20:35.112507 5131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:35.104491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.62:42235 (local address 127.31.100.1:49843)
0812 05:20:35.104648 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0812 05:20:35.104685 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0812 05:20:35.105144 (+ 459us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:20:35.105334 (+ 190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:20:35.105337 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:20:35.105903 (+ 566us) client_negotiation.cc:189] Negotiated authn=SASL
0812 05:20:35.106090 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:20:35.106095 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:20:35.106728 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:20:35.106729 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:20:35.107532 (+ 803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:20:35.107534 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:20:35.107851 (+ 317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:20:35.107852 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:20:35.107952 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:35.108356 (+ 404us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 05:20:35.108364 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0812 05:20:35.109725 (+ 1361us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 05:20:35.111124 (+ 1399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 05:20:35.111128 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 05:20:35.111130 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 05:20:35.111325 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 05:20:35.111500 (+ 175us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 05:20:35.111501 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 05:20:35.111502 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 05:20:35.111611 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 05:20:35.111882 (+ 271us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 05:20:35.111891 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 05:20:35.112044 (+ 153us) client_negotiation.cc:769] Sending connection context
0812 05:20:35.112139 (+ 95us) client_negotiation.cc:240] Negotiation successful
0812 05:20:35.112301 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":81,"threads_started":1}
I20260812 05:20:35.112917 5128 heartbeater.cc:347] Connected to a master server at 127.31.100.62:42235
I20260812 05:20:35.113019 5128 heartbeater.cc:464] Registering TS with master...
I20260812 05:20:35.113188 5128 heartbeater.cc:511] Master 127.31.100.62:42235 requested a full tablet report, sending...
I20260812 05:20:35.113684 4804 ts_manager.cc:194] Registered new tserver with Master: 43863ed5260a43f884343e8c47e77e42 (127.31.100.1:35107)
I20260812 05:20:35.114260 4804 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.100.1@KRBTEST.COM'} at 127.31.100.1:49843
WARNING: no policy specified for kudu/127.31.100.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.100.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.100.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.2.keytab.
Entry for principal kudu/127.31.100.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.2.keytab.
I20260812 05:20:35.135970 32144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0waFvT/build/release/bin/kudu
/tmp/dist-test-task0waFvT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786512017613366-32144-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786512017613366-32144-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786512017613366-32144-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786512017613366-32144-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.100.2:0
--local_ip_for_outbound_sockets=127.31.100.2
--webserver_interface=127.31.100.2
--webserver_port=0
--tserver_master_addrs=127.31.100.62:42235
--keytab_file=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.2.keytab
--principal=kudu/127.31.100.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.100.20:38789
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786512017613366-32144-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786512017613366-32144-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786512017613366-32144-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 05:20:35.217628 5138 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 05:20:35.217811 5138 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 05:20:35.217833 5138 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 05:20:35.218879 5138 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 05:20:35.218911 5138 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 05:20:35.218936 5138 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.100.2
I20260812 05:20:35.220350 5138 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.100.20:38789
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786512017613366-32144-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786512017613366-32144-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.2.keytab
--principal=kudu/127.31.100.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.100.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786512017613366-32144-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.100.2
--webserver_port=0
--tserver_master_addrs=127.31.100.62:42235
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5138
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.100.2
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786512017613366-32144-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 04:20:34 UTC on ccycloud.quasar-phqdqp.root.comops.site
build id 4581871
I20260812 05:20:35.220547 5138 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 05:20:35.220727 5138 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260812 05:20:35.225390 5138 server_base.cc:1034] running on GCE node
W20260812 05:20:35.225346 5147 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 05:20:35.225353 5150 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 05:20:35.225329 5146 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 05:20:35.225783 5138 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 05:20:35.226027 5138 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 05:20:35.227171 5138 hybrid_clock.cc:648] HybridClock initialized: now 1786512035227156 us; error 28 us; skew 500 ppm
Aug 12 05:20:35 dist-test-slave-42z9 krb5kdc[4739](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512035, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.100.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 05:20:35.229600 5138 init.cc:377] Logged in from keytab as kudu/127.31.100.2@KRBTEST.COM (short username kudu)
I20260812 05:20:35.230002 5138 webserver.cc:460] Webserver started at http://127.31.100.2:37215/ using document root <none> and password file <none>
I20260812 05:20:35.230242 5138 fs_manager.cc:359] Metadata directory not provided
I20260812 05:20:35.230300 5138 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 05:20:35.230425 5138 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 05:20:35.231396 5138 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786512017613366-32144-0/minicluster-data/ts-1/data/instance:
uuid: "303ff3e5d4b44362a7a55ccc40c7d352"
format_stamp: "Formatted at 2026-08-12 05:20:35 on dist-test-slave-42z9"
server_key: "21c8e343a9277684572ee2791ec1ad84"
server_key_iv: "e2dc47b2cd4f2cac1e790dc32d6ebaa7"
server_key_version: "encryptionkey@0"
I20260812 05:20:35.231719 5138 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786512017613366-32144-0/minicluster-data/ts-1/wal/instance:
uuid: "303ff3e5d4b44362a7a55ccc40c7d352"
format_stamp: "Formatted at 2026-08-12 05:20:35 on dist-test-slave-42z9"
server_key: "21c8e343a9277684572ee2791ec1ad84"
server_key_iv: "e2dc47b2cd4f2cac1e790dc32d6ebaa7"
server_key_version: "encryptionkey@0"
I20260812 05:20:35.232937 5138 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260812 05:20:35.233631 5165 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 05:20:35.233767 5138 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260812 05:20:35.233858 5138 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786512017613366-32144-0/minicluster-data/ts-1/data,/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786512017613366-32144-0/minicluster-data/ts-1/wal
uuid: "303ff3e5d4b44362a7a55ccc40c7d352"
format_stamp: "Formatted at 2026-08-12 05:20:35 on dist-test-slave-42z9"
server_key: "21c8e343a9277684572ee2791ec1ad84"
server_key_iv: "e2dc47b2cd4f2cac1e790dc32d6ebaa7"
server_key_version: "encryptionkey@0"
I20260812 05:20:35.233927 5138 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786512017613366-32144-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786512017613366-32144-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786512017613366-32144-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 05:20:35.242372 5138 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 05:20:35.244031 5138 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 05:20:35.244155 5138 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 05:20:35.244350 5138 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 05:20:35.244637 5138 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 05:20:35.244669 5138 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 05:20:35.244697 5138 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 05:20:35.244720 5138 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 05:20:35.250388 5138 rpc_server.cc:307] RPC server started. Bound to: 127.31.100.2:40735
I20260812 05:20:35.250432 5345 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.100.2:40735 every 8 connection(s)
I20260812 05:20:35.250747 5138 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786512017613366-32144-0/minicluster-data/ts-1/data/info.pb
Aug 12 05:20:35 dist-test-slave-42z9 krb5kdc[4739](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512035, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.100.2@KRBTEST.COM for kudu/127.31.100.62@KRBTEST.COM
I20260812 05:20:35.259476 4906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:35.251398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.100.2:41315 (local address 127.31.100.62:42235)
0812 05:20:35.251454 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0812 05:20:35.251455 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0812 05:20:35.251659 (+ 204us) server_negotiation.cc:407] Connection header received
0812 05:20:35.252127 (+ 468us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:20:35.252128 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:20:35.252137 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:20:35.252190 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:20:35.253126 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:35.253727 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:35.254729 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:35.254938 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:35.254994 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:35.256901 (+ 1907us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 05:20:35.256902 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 05:20:35.256903 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 05:20:35.256910 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 05:20:35.258265 (+ 1355us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:20:35.258623 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:20:35.258623 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:20:35.258624 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:20:35.258679 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:20:35.258932 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:20:35.258933 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:20:35.258934 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:20:35.259048 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 05:20:35.259106 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0812 05:20:35.259321 (+ 215us) server_negotiation.cc:299] Negotiation successful
0812 05:20:35.259396 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260812 05:20:35.259663 5350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:35.251378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.62:42235 (local address 127.31.100.2:41315)
0812 05:20:35.251542 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0812 05:20:35.251583 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0812 05:20:35.252051 (+ 468us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:20:35.252244 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:20:35.252247 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:20:35.252820 (+ 573us) client_negotiation.cc:189] Negotiated authn=SASL
0812 05:20:35.253010 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:20:35.253015 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:20:35.253822 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:20:35.253822 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:20:35.254648 (+ 826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:20:35.254651 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:20:35.254999 (+ 348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:20:35.255000 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:20:35.255099 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:35.255553 (+ 454us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 05:20:35.255561 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0812 05:20:35.256821 (+ 1260us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 05:20:35.258339 (+ 1518us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 05:20:35.258346 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 05:20:35.258347 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 05:20:35.258565 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 05:20:35.258728 (+ 163us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 05:20:35.258729 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 05:20:35.258729 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 05:20:35.258840 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 05:20:35.259103 (+ 263us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 05:20:35.259106 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 05:20:35.259224 (+ 118us) client_negotiation.cc:769] Sending connection context
0812 05:20:35.259302 (+ 78us) client_negotiation.cc:240] Negotiation successful
0812 05:20:35.259429 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":76,"threads_started":1}
I20260812 05:20:35.260102 5346 heartbeater.cc:347] Connected to a master server at 127.31.100.62:42235
I20260812 05:20:35.260191 5346 heartbeater.cc:464] Registering TS with master...
I20260812 05:20:35.260360 5346 heartbeater.cc:511] Master 127.31.100.62:42235 requested a full tablet report, sending...
I20260812 05:20:35.260725 32144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0waFvT/build/release/bin/kudu as pid 5138
I20260812 05:20:35.260771 4804 ts_manager.cc:194] Registered new tserver with Master: 303ff3e5d4b44362a7a55ccc40c7d352 (127.31.100.2:40735)
I20260812 05:20:35.260828 32144 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786512017613366-32144-0/minicluster-data/ts-1/wal/instance
I20260812 05:20:35.261039 32144 external_mini_cluster.cc:1428] Setting key 0be2c969830d5cae7d04c85334eb87ae
I20260812 05:20:35.261245 4804 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.100.2@KRBTEST.COM'} at 127.31.100.2:41315
WARNING: no policy specified for kudu/127.31.100.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.100.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.100.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.3.keytab.
Entry for principal kudu/127.31.100.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.3.keytab.
I20260812 05:20:35.285434 32144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0waFvT/build/release/bin/kudu
/tmp/dist-test-task0waFvT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786512017613366-32144-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786512017613366-32144-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786512017613366-32144-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786512017613366-32144-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.100.3:0
--local_ip_for_outbound_sockets=127.31.100.3
--webserver_interface=127.31.100.3
--webserver_port=0
--tserver_master_addrs=127.31.100.62:42235
--keytab_file=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.3.keytab
--principal=kudu/127.31.100.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.100.20:38789
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786512017613366-32144-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786512017613366-32144-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786512017613366-32144-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 05:20:35.367573 5353 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 05:20:35.367772 5353 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 05:20:35.367794 5353 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 05:20:35.368901 5353 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 05:20:35.368943 5353 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 05:20:35.368971 5353 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.100.3
I20260812 05:20:35.370455 5353 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.100.20:38789
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786512017613366-32144-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786512017613366-32144-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.3.keytab
--principal=kudu/127.31.100.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.100.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786512017613366-32144-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.100.3
--webserver_port=0
--tserver_master_addrs=127.31.100.62:42235
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5353
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.100.3
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786512017613366-32144-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 04:20:34 UTC on ccycloud.quasar-phqdqp.root.comops.site
build id 4581871
I20260812 05:20:35.370685 5353 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 05:20:35.370888 5353 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 05:20:35.375679 5370 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 05:20:35.375713 5361 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 05:20:35.375876 5360 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 05:20:35.375720 5353 server_base.cc:1034] running on GCE node
I20260812 05:20:35.376072 5353 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 05:20:35.376288 5353 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 05:20:35.377455 5353 hybrid_clock.cc:648] HybridClock initialized: now 1786512035377417 us; error 31 us; skew 500 ppm
Aug 12 05:20:35 dist-test-slave-42z9 krb5kdc[4739](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512035, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.100.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 05:20:35.379956 5353 init.cc:377] Logged in from keytab as kudu/127.31.100.3@KRBTEST.COM (short username kudu)
I20260812 05:20:35.380384 5353 webserver.cc:460] Webserver started at http://127.31.100.3:38333/ using document root <none> and password file <none>
I20260812 05:20:35.380589 5353 fs_manager.cc:359] Metadata directory not provided
I20260812 05:20:35.380628 5353 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 05:20:35.380719 5353 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 05:20:35.381742 5353 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786512017613366-32144-0/minicluster-data/ts-2/data/instance:
uuid: "94aea65ca2514a3ca70a64a633fa5dd4"
format_stamp: "Formatted at 2026-08-12 05:20:35 on dist-test-slave-42z9"
server_key: "c602b9c8f84a95d34997f22020900428"
server_key_iv: "548bbe26d1ae3d83b47caf4838b9c234"
server_key_version: "encryptionkey@0"
I20260812 05:20:35.382031 5353 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786512017613366-32144-0/minicluster-data/ts-2/wal/instance:
uuid: "94aea65ca2514a3ca70a64a633fa5dd4"
format_stamp: "Formatted at 2026-08-12 05:20:35 on dist-test-slave-42z9"
server_key: "c602b9c8f84a95d34997f22020900428"
server_key_iv: "548bbe26d1ae3d83b47caf4838b9c234"
server_key_version: "encryptionkey@0"
I20260812 05:20:35.383170 5353 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260812 05:20:35.383960 5378 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 05:20:35.384157 5353 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 05:20:35.384254 5353 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786512017613366-32144-0/minicluster-data/ts-2/data,/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786512017613366-32144-0/minicluster-data/ts-2/wal
uuid: "94aea65ca2514a3ca70a64a633fa5dd4"
format_stamp: "Formatted at 2026-08-12 05:20:35 on dist-test-slave-42z9"
server_key: "c602b9c8f84a95d34997f22020900428"
server_key_iv: "548bbe26d1ae3d83b47caf4838b9c234"
server_key_version: "encryptionkey@0"
I20260812 05:20:35.384311 5353 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786512017613366-32144-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786512017613366-32144-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786512017613366-32144-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 05:20:35.418612 5353 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 05:20:35.420843 5353 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 05:20:35.420962 5353 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 05:20:35.421162 5353 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 05:20:35.421458 5353 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 05:20:35.421504 5353 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 05:20:35.421527 5353 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 05:20:35.421543 5353 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 05:20:35.427137 5353 rpc_server.cc:307] RPC server started. Bound to: 127.31.100.3:35081
I20260812 05:20:35.427192 5568 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.100.3:35081 every 8 connection(s)
I20260812 05:20:35.427495 5353 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786512017613366-32144-0/minicluster-data/ts-2/data/info.pb
I20260812 05:20:35.430521 32144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0waFvT/build/release/bin/kudu as pid 5353
I20260812 05:20:35.430624 32144 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786512017613366-32144-0/minicluster-data/ts-2/wal/instance
I20260812 05:20:35.430804 32144 external_mini_cluster.cc:1428] Setting key ec2893e2d260bff963bdd80a0aba2e02
Aug 12 05:20:35 dist-test-slave-42z9 krb5kdc[4739](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512035, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.100.3@KRBTEST.COM for kudu/127.31.100.62@KRBTEST.COM
I20260812 05:20:35.436247 4906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:35.428139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.100.3:45291 (local address 127.31.100.62:42235)
0812 05:20:35.428226 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0812 05:20:35.428228 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 05:20:35.428424 (+ 196us) server_negotiation.cc:407] Connection header received
0812 05:20:35.428886 (+ 462us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:20:35.428887 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:20:35.428898 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:20:35.428955 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:20:35.429863 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:35.430377 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:35.431686 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:35.431873 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:35.431922 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:35.433836 (+ 1914us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 05:20:35.433837 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 05:20:35.433838 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 05:20:35.433844 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0812 05:20:35.435036 (+ 1192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:20:35.435429 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:20:35.435430 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:20:35.435430 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:20:35.435491 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:20:35.435712 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:20:35.435712 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:20:35.435713 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:20:35.435815 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 05:20:35.435861 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0812 05:20:35.436113 (+ 252us) server_negotiation.cc:299] Negotiation successful
0812 05:20:35.436180 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260812 05:20:35.436434 5576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:35.428115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.62:42235 (local address 127.31.100.3:45291)
0812 05:20:35.428279 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0812 05:20:35.428315 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0812 05:20:35.428808 (+ 493us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:20:35.429010 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:20:35.429013 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:20:35.429589 (+ 576us) client_negotiation.cc:189] Negotiated authn=SASL
0812 05:20:35.429782 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:20:35.429786 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:20:35.430458 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:20:35.430459 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:20:35.431615 (+ 1156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:20:35.431618 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:20:35.431930 (+ 312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:20:35.431931 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:20:35.432037 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:35.432466 (+ 429us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 05:20:35.432474 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0812 05:20:35.433758 (+ 1284us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 05:20:35.435125 (+ 1367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 05:20:35.435131 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 05:20:35.435132 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 05:20:35.435338 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 05:20:35.435555 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 05:20:35.435556 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 05:20:35.435556 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 05:20:35.435664 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 05:20:35.435875 (+ 211us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 05:20:35.435880 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 05:20:35.435999 (+ 119us) client_negotiation.cc:769] Sending connection context
0812 05:20:35.436082 (+ 83us) client_negotiation.cc:240] Negotiation successful
0812 05:20:35.436195 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":85,"threads_started":1}
I20260812 05:20:35.436902 5571 heartbeater.cc:347] Connected to a master server at 127.31.100.62:42235
I20260812 05:20:35.436991 5571 heartbeater.cc:464] Registering TS with master...
I20260812 05:20:35.437150 5571 heartbeater.cc:511] Master 127.31.100.62:42235 requested a full tablet report, sending...
I20260812 05:20:35.437546 4803 ts_manager.cc:194] Registered new tserver with Master: 94aea65ca2514a3ca70a64a633fa5dd4 (127.31.100.3:35081)
I20260812 05:20:35.437920 4803 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.100.3@KRBTEST.COM'} at 127.31.100.3:45291
I20260812 05:20:35.442302 32144 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 12 05:20:35 dist-test-slave-42z9 krb5kdc[4739](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512035, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Aug 12 05:20:35 dist-test-slave-42z9 krb5kdc[4739](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512035, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.31.100.62@KRBTEST.COM
I20260812 05:20:35.465282 4906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:35.458342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34324 (local address 127.31.100.62:42235)
0812 05:20:35.458410 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0812 05:20:35.458413 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 05:20:35.458509 (+ 96us) server_negotiation.cc:407] Connection header received
0812 05:20:35.458588 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:20:35.458589 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:20:35.458600 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:20:35.458647 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:20:35.459224 (+ 577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:35.459791 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:35.460532 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:35.460721 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:35.460774 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:35.462921 (+ 2147us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 05:20:35.462922 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 05:20:35.462923 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 05:20:35.462930 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 05:20:35.464201 (+ 1271us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:20:35.464533 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:20:35.464534 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:20:35.464535 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:20:35.464587 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:20:35.464745 (+ 158us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:20:35.464746 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:20:35.464746 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:20:35.464891 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 05:20:35.464982 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0812 05:20:35.465107 (+ 125us) server_negotiation.cc:299] Negotiation successful
0812 05:20:35.465185 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
W20260812 05:20:35.465555 4803 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:34324
I20260812 05:20:35.466050 32144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0waFvT/build/release/bin/kudu with pid 4912
I20260812 05:20:35.479764 32144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0waFvT/build/release/bin/kudu with pid 5138
I20260812 05:20:35.493152 32144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0waFvT/build/release/bin/kudu with pid 5353
I20260812 05:20:35.500592 32144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0waFvT/build/release/bin/kudu with pid 4759
2026-08-12T05:20:35Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (2028 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786512017613366-32144-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 05:20:35 dist-test-slave-42z9 krb5kdc[5592](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 05:20:35 dist-test-slave-42z9 krb5kdc[5592](info): set up 2 sockets
Aug 12 05:20:35 dist-test-slave-42z9 krb5kdc[5592](info): commencing operation
krb5kdc: starting...
W20260812 05:20:36.677673 32144 mini_kdc.cc:121] Time spent starting KDC: real 1.146s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786512017613366-32144-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786512017613366-32144-0/krb5kdc/test-user.keytab.
Aug 12 05:20:36 dist-test-slave-42z9 krb5kdc[5592](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512036, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T05:20:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T05:20:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.100.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.100.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.100.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.62.keytab.
Entry for principal kudu/127.31.100.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.62.keytab.
I20260812 05:20:36.792565 32144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0waFvT/build/release/bin/kudu
/tmp/dist-test-task0waFvT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786512017613366-32144-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786512017613366-32144-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786512017613366-32144-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786512017613366-32144-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.100.62:40701
--webserver_interface=127.31.100.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.62.keytab
--principal=kudu/127.31.100.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.100.20:34225
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.100.62:40701
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786512017613366-32144-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786512017613366-32144-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786512017613366-32144-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 05:20:36.871683 5611 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 05:20:36.871872 5611 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 05:20:36.871902 5611 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 05:20:36.873021 5611 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 05:20:36.873059 5611 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 05:20:36.873080 5611 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 05:20:36.873095 5611 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 05:20:36.873109 5611 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 05:20:36.874604 5611 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.100.20:34225
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786512017613366-32144-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786512017613366-32144-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.100.62:40701
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.62.keytab
--principal=kudu/127.31.100.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.100.62:40701
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786512017613366-32144-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.100.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5611
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786512017613366-32144-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 04:20:34 UTC on ccycloud.quasar-phqdqp.root.comops.site
build id 4581871
I20260812 05:20:36.874840 5611 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 05:20:36.875030 5611 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 05:20:36.879696 5623 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 05:20:36.879706 5621 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 05:20:36.879706 5619 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 05:20:36.879841 5611 server_base.cc:1034] running on GCE node
I20260812 05:20:36.880091 5611 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 05:20:36.880339 5611 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 05:20:36.881492 5611 hybrid_clock.cc:648] HybridClock initialized: now 1786512036881476 us; error 35 us; skew 500 ppm
Aug 12 05:20:36 dist-test-slave-42z9 krb5kdc[5592](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512036, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.100.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 05:20:36.883790 5611 init.cc:377] Logged in from keytab as kudu/127.31.100.62@KRBTEST.COM (short username kudu)
I20260812 05:20:36.884202 5611 webserver.cc:460] Webserver started at http://127.31.100.62:41567/ using document root <none> and password file <none>
I20260812 05:20:36.884371 5611 fs_manager.cc:359] Metadata directory not provided
I20260812 05:20:36.884409 5611 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 05:20:36.884496 5611 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 05:20:36.885507 5611 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786512017613366-32144-0/minicluster-data/master-0/data/instance:
uuid: "c8d88bbb5271464d9f125b9e9135c171"
format_stamp: "Formatted at 2026-08-12 05:20:36 on dist-test-slave-42z9"
server_key: "7bbd2de03a72e87f4a40508078ddd897"
server_key_iv: "f014bee74eb341b1dd15c0c1209db560"
server_key_version: "encryptionkey@0"
I20260812 05:20:36.885805 5611 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786512017613366-32144-0/minicluster-data/master-0/wal/instance:
uuid: "c8d88bbb5271464d9f125b9e9135c171"
format_stamp: "Formatted at 2026-08-12 05:20:36 on dist-test-slave-42z9"
server_key: "7bbd2de03a72e87f4a40508078ddd897"
server_key_iv: "f014bee74eb341b1dd15c0c1209db560"
server_key_version: "encryptionkey@0"
I20260812 05:20:36.886956 5611 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260812 05:20:36.887564 5632 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 05:20:36.887732 5611 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260812 05:20:36.887817 5611 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786512017613366-32144-0/minicluster-data/master-0/data,/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786512017613366-32144-0/minicluster-data/master-0/wal
uuid: "c8d88bbb5271464d9f125b9e9135c171"
format_stamp: "Formatted at 2026-08-12 05:20:36 on dist-test-slave-42z9"
server_key: "7bbd2de03a72e87f4a40508078ddd897"
server_key_iv: "f014bee74eb341b1dd15c0c1209db560"
server_key_version: "encryptionkey@0"
I20260812 05:20:36.887869 5611 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786512017613366-32144-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786512017613366-32144-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786512017613366-32144-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 05:20:36.911640 5611 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 05:20:36.913760 5611 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 05:20:36.913869 5611 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 05:20:36.917701 5611 rpc_server.cc:307] RPC server started. Bound to: 127.31.100.62:40701
I20260812 05:20:36.917929 5728 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.100.62:40701 every 8 connection(s)
I20260812 05:20:36.918061 5611 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786512017613366-32144-0/minicluster-data/master-0/data/info.pb
I20260812 05:20:36.918742 5731 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 05:20:36.920985 5731 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c8d88bbb5271464d9f125b9e9135c171: Bootstrap starting.
I20260812 05:20:36.921562 5731 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c8d88bbb5271464d9f125b9e9135c171: Neither blocks nor log segments found. Creating new log.
I20260812 05:20:36.921838 5731 log.cc:824] T 00000000000000000000000000000000 P c8d88bbb5271464d9f125b9e9135c171: Log is configured to *not* fsync() on all Append() calls
I20260812 05:20:36.922492 5731 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c8d88bbb5271464d9f125b9e9135c171: No bootstrap required, opened a new log
I20260812 05:20:36.924193 5731 raft_consensus.cc:348] T 00000000000000000000000000000000 P c8d88bbb5271464d9f125b9e9135c171 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8d88bbb5271464d9f125b9e9135c171" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 40701 } }
I20260812 05:20:36.924325 5731 raft_consensus.cc:374] T 00000000000000000000000000000000 P c8d88bbb5271464d9f125b9e9135c171 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 05:20:36.924356 5731 raft_consensus.cc:729] T 00000000000000000000000000000000 P c8d88bbb5271464d9f125b9e9135c171 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8d88bbb5271464d9f125b9e9135c171, State: Initialized, Role: FOLLOWER
I20260812 05:20:36.924484 5731 consensus_queue.cc:260] T 00000000000000000000000000000000 P c8d88bbb5271464d9f125b9e9135c171 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8d88bbb5271464d9f125b9e9135c171" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 40701 } }
I20260812 05:20:36.924546 5731 raft_consensus.cc:388] T 00000000000000000000000000000000 P c8d88bbb5271464d9f125b9e9135c171 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 05:20:36.924576 5731 raft_consensus.cc:482] T 00000000000000000000000000000000 P c8d88bbb5271464d9f125b9e9135c171 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 05:20:36.924621 5731 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c8d88bbb5271464d9f125b9e9135c171 [term 0 FOLLOWER]: Advancing to term 1
I20260812 05:20:36.925400 5731 raft_consensus.cc:504] T 00000000000000000000000000000000 P c8d88bbb5271464d9f125b9e9135c171 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8d88bbb5271464d9f125b9e9135c171" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 40701 } }
I20260812 05:20:36.925522 5731 leader_election.cc:302] T 00000000000000000000000000000000 P c8d88bbb5271464d9f125b9e9135c171 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c8d88bbb5271464d9f125b9e9135c171; no voters:
I20260812 05:20:36.925680 5731 leader_election.cc:288] T 00000000000000000000000000000000 P c8d88bbb5271464d9f125b9e9135c171 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 05:20:36.925729 5736 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c8d88bbb5271464d9f125b9e9135c171 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 05:20:36.925879 5736 raft_consensus.cc:686] T 00000000000000000000000000000000 P c8d88bbb5271464d9f125b9e9135c171 [term 1 LEADER]: Becoming Leader. State: Replica: c8d88bbb5271464d9f125b9e9135c171, State: Running, Role: LEADER
I20260812 05:20:36.925917 5731 sys_catalog.cc:564] T 00000000000000000000000000000000 P c8d88bbb5271464d9f125b9e9135c171 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 05:20:36.925994 5736 consensus_queue.cc:237] T 00000000000000000000000000000000 P c8d88bbb5271464d9f125b9e9135c171 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8d88bbb5271464d9f125b9e9135c171" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 40701 } }
I20260812 05:20:36.926333 5738 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8d88bbb5271464d9f125b9e9135c171 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c8d88bbb5271464d9f125b9e9135c171. Latest consensus state: current_term: 1 leader_uuid: "c8d88bbb5271464d9f125b9e9135c171" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8d88bbb5271464d9f125b9e9135c171" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 40701 } } }
I20260812 05:20:36.926425 5738 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8d88bbb5271464d9f125b9e9135c171 [sys.catalog]: This master's current role is: LEADER
I20260812 05:20:36.926322 5737 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8d88bbb5271464d9f125b9e9135c171 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c8d88bbb5271464d9f125b9e9135c171" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8d88bbb5271464d9f125b9e9135c171" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 40701 } } }
I20260812 05:20:36.926564 5737 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8d88bbb5271464d9f125b9e9135c171 [sys.catalog]: This master's current role is: LEADER
I20260812 05:20:36.926687 5751 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 05:20:36.927110 32144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0waFvT/build/release/bin/kudu as pid 5611
I20260812 05:20:36.927219 32144 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786512017613366-32144-0/minicluster-data/master-0/wal/instance
I20260812 05:20:36.927270 5751 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 05:20:36.927423 32144 external_mini_cluster.cc:1428] Setting key 519707ca1058c255606a7aaa52f7f2bd
I20260812 05:20:36.928803 5751 catalog_manager.cc:1347] Generated new cluster ID: 9dac854003e441e58956f9a629e5ea9a
I20260812 05:20:36.928856 5751 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 12 05:20:36 dist-test-slave-42z9 krb5kdc[5592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512036, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.100.62@KRBTEST.COM
I20260812 05:20:36.936184 5768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:36.927853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42760 (local address 127.31.100.62:40701)
0812 05:20:36.927999 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0812 05:20:36.928007 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 05:20:36.928070 (+ 63us) server_negotiation.cc:407] Connection header received
0812 05:20:36.928173 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:20:36.928180 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:20:36.928231 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:20:36.928364 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:20:36.929055 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:36.930013 (+ 958us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:36.930716 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:36.931178 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:36.931269 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:36.933340 (+ 2071us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 05:20:36.933344 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 05:20:36.933344 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 05:20:36.933356 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0812 05:20:36.934829 (+ 1473us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:20:36.935136 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:20:36.935139 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:20:36.935139 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:20:36.935229 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:20:36.935397 (+ 168us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:20:36.935397 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:20:36.935398 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:20:36.935619 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 05:20:36.935766 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0812 05:20:36.935858 (+ 92us) server_negotiation.cc:299] Negotiation successful
0812 05:20:36.935999 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101,"thread_start_us":75,"threads_started":1}
I20260812 05:20:36.940093 5751 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 05:20:36.940763 5751 catalog_manager.cc:1504] Loading token signing keys...
I20260812 05:20:36.947804 5751 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c8d88bbb5271464d9f125b9e9135c171: Generated new TSK 0
I20260812 05:20:36.947962 5751 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.31.100.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.100.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.100.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.1.keytab.
Entry for principal kudu/127.31.100.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.1.keytab.
I20260812 05:20:36.983537 32144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0waFvT/build/release/bin/kudu
/tmp/dist-test-task0waFvT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786512017613366-32144-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786512017613366-32144-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786512017613366-32144-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786512017613366-32144-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.100.1:0
--local_ip_for_outbound_sockets=127.31.100.1
--webserver_interface=127.31.100.1
--webserver_port=0
--tserver_master_addrs=127.31.100.62:40701
--keytab_file=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.1.keytab
--principal=kudu/127.31.100.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.100.20:34225
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786512017613366-32144-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786512017613366-32144-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786512017613366-32144-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 05:20:37.063925 5777 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 05:20:37.064131 5777 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 05:20:37.064152 5777 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 05:20:37.065274 5777 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 05:20:37.065312 5777 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 05:20:37.065337 5777 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.100.1
I20260812 05:20:37.066874 5777 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.100.20:34225
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786512017613366-32144-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786512017613366-32144-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.1.keytab
--principal=kudu/127.31.100.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.100.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786512017613366-32144-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.100.1
--webserver_port=0
--tserver_master_addrs=127.31.100.62:40701
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5777
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.100.1
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786512017613366-32144-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 04:20:34 UTC on ccycloud.quasar-phqdqp.root.comops.site
build id 4581871
I20260812 05:20:37.067086 5777 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 05:20:37.067294 5777 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 05:20:37.072094 5784 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 05:20:37.072091 5785 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 05:20:37.072118 5788 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 05:20:37.072389 5777 server_base.cc:1034] running on GCE node
I20260812 05:20:37.072592 5777 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 05:20:37.072818 5777 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 05:20:37.073998 5777 hybrid_clock.cc:648] HybridClock initialized: now 1786512037073982 us; error 29 us; skew 500 ppm
Aug 12 05:20:37 dist-test-slave-42z9 krb5kdc[5592](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512037, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.100.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 05:20:37.076553 5777 init.cc:377] Logged in from keytab as kudu/127.31.100.1@KRBTEST.COM (short username kudu)
I20260812 05:20:37.076987 5777 webserver.cc:460] Webserver started at http://127.31.100.1:38725/ using document root <none> and password file <none>
I20260812 05:20:37.077178 5777 fs_manager.cc:359] Metadata directory not provided
I20260812 05:20:37.077219 5777 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 05:20:37.077338 5777 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 05:20:37.078416 5777 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786512017613366-32144-0/minicluster-data/ts-0/data/instance:
uuid: "7b11aabe1b1c478a9a87a263bd78fae1"
format_stamp: "Formatted at 2026-08-12 05:20:37 on dist-test-slave-42z9"
server_key: "cd04091afbce76fc19d4ae50e75c9f06"
server_key_iv: "0c7e48c42d212c1f3433b0140286ac15"
server_key_version: "encryptionkey@0"
I20260812 05:20:37.078730 5777 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786512017613366-32144-0/minicluster-data/ts-0/wal/instance:
uuid: "7b11aabe1b1c478a9a87a263bd78fae1"
format_stamp: "Formatted at 2026-08-12 05:20:37 on dist-test-slave-42z9"
server_key: "cd04091afbce76fc19d4ae50e75c9f06"
server_key_iv: "0c7e48c42d212c1f3433b0140286ac15"
server_key_version: "encryptionkey@0"
I20260812 05:20:37.079979 5777 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260812 05:20:37.080619 5803 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 05:20:37.080762 5777 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260812 05:20:37.080842 5777 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786512017613366-32144-0/minicluster-data/ts-0/data,/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786512017613366-32144-0/minicluster-data/ts-0/wal
uuid: "7b11aabe1b1c478a9a87a263bd78fae1"
format_stamp: "Formatted at 2026-08-12 05:20:37 on dist-test-slave-42z9"
server_key: "cd04091afbce76fc19d4ae50e75c9f06"
server_key_iv: "0c7e48c42d212c1f3433b0140286ac15"
server_key_version: "encryptionkey@0"
I20260812 05:20:37.080904 5777 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786512017613366-32144-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786512017613366-32144-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786512017613366-32144-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 05:20:37.097434 5777 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 05:20:37.099496 5777 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 05:20:37.099622 5777 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 05:20:37.099833 5777 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 05:20:37.100123 5777 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 05:20:37.100155 5777 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 05:20:37.100178 5777 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 05:20:37.100199 5777 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 05:20:37.105865 5777 rpc_server.cc:307] RPC server started. Bound to: 127.31.100.1:38345
I20260812 05:20:37.105903 5982 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.100.1:38345 every 8 connection(s)
I20260812 05:20:37.106215 5777 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786512017613366-32144-0/minicluster-data/ts-0/data/info.pb
I20260812 05:20:37.108193 32144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0waFvT/build/release/bin/kudu as pid 5777
I20260812 05:20:37.108316 32144 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786512017613366-32144-0/minicluster-data/ts-0/wal/instance
I20260812 05:20:37.108526 32144 external_mini_cluster.cc:1428] Setting key e72e2330d1e45cd633fe847acd76b52c
Aug 12 05:20:37 dist-test-slave-42z9 krb5kdc[5592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512037, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.100.1@KRBTEST.COM for kudu/127.31.100.62@KRBTEST.COM
I20260812 05:20:37.115736 5768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:37.106868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.100.1:42843 (local address 127.31.100.62:40701)
0812 05:20:37.106951 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0812 05:20:37.106953 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 05:20:37.107107 (+ 154us) server_negotiation.cc:407] Connection header received
0812 05:20:37.108014 (+ 907us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:20:37.108015 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:20:37.108029 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:20:37.108082 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:20:37.108849 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:37.109492 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:37.110456 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:37.110742 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:37.110800 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:37.112897 (+ 2097us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 05:20:37.112898 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 05:20:37.112899 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 05:20:37.112907 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0812 05:20:37.114505 (+ 1598us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:20:37.114854 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:20:37.114855 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:20:37.114856 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:20:37.114917 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:20:37.115122 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:20:37.115123 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:20:37.115123 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:20:37.115251 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 05:20:37.115316 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0812 05:20:37.115542 (+ 226us) server_negotiation.cc:299] Negotiation successful
0812 05:20:37.115640 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260812 05:20:37.115767 5985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:37.106839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.62:40701 (local address 127.31.100.1:42843)
0812 05:20:37.106995 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0812 05:20:37.107031 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0812 05:20:37.107914 (+ 883us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:20:37.108159 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:20:37.108163 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:20:37.108507 (+ 344us) client_negotiation.cc:189] Negotiated authn=SASL
0812 05:20:37.108759 (+ 252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:20:37.108765 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:20:37.109581 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:20:37.109582 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:20:37.110371 (+ 789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:20:37.110374 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:20:37.110832 (+ 458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:20:37.110833 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:20:37.110972 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:37.111383 (+ 411us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 05:20:37.111392 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0812 05:20:37.112803 (+ 1411us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 05:20:37.114580 (+ 1777us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 05:20:37.114586 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 05:20:37.114588 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 05:20:37.114789 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 05:20:37.114976 (+ 187us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 05:20:37.114976 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 05:20:37.114976 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 05:20:37.115068 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 05:20:37.115310 (+ 242us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 05:20:37.115314 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 05:20:37.115419 (+ 105us) client_negotiation.cc:769] Sending connection context
0812 05:20:37.115504 (+ 85us) client_negotiation.cc:240] Negotiation successful
0812 05:20:37.115618 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":85,"threads_started":1}
I20260812 05:20:37.116178 5983 heartbeater.cc:347] Connected to a master server at 127.31.100.62:40701
I20260812 05:20:37.116276 5983 heartbeater.cc:464] Registering TS with master...
I20260812 05:20:37.116477 5983 heartbeater.cc:511] Master 127.31.100.62:40701 requested a full tablet report, sending...
I20260812 05:20:37.116981 5669 ts_manager.cc:194] Registered new tserver with Master: 7b11aabe1b1c478a9a87a263bd78fae1 (127.31.100.1:38345)
WARNING: no policy specified for kudu/127.31.100.2@KRBTEST.COM; defaulting to no policy
I20260812 05:20:37.117650 5669 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.100.1@KRBTEST.COM'} at 127.31.100.1:42843
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.100.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.100.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.2.keytab.
Entry for principal kudu/127.31.100.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.2.keytab.
I20260812 05:20:37.133121 32144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0waFvT/build/release/bin/kudu
/tmp/dist-test-task0waFvT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786512017613366-32144-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786512017613366-32144-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786512017613366-32144-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786512017613366-32144-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.100.2:0
--local_ip_for_outbound_sockets=127.31.100.2
--webserver_interface=127.31.100.2
--webserver_port=0
--tserver_master_addrs=127.31.100.62:40701
--keytab_file=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.2.keytab
--principal=kudu/127.31.100.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.100.20:34225
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786512017613366-32144-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786512017613366-32144-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786512017613366-32144-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 05:20:37.216593 5989 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 05:20:37.216778 5989 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 05:20:37.216809 5989 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 05:20:37.217942 5989 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 05:20:37.217983 5989 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 05:20:37.218016 5989 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.100.2
I20260812 05:20:37.219496 5989 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.100.20:34225
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786512017613366-32144-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786512017613366-32144-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.2.keytab
--principal=kudu/127.31.100.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.100.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786512017613366-32144-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.100.2
--webserver_port=0
--tserver_master_addrs=127.31.100.62:40701
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5989
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.100.2
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786512017613366-32144-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 04:20:34 UTC on ccycloud.quasar-phqdqp.root.comops.site
build id 4581871
I20260812 05:20:37.219709 5989 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 05:20:37.219904 5989 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 05:20:37.224589 5997 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 05:20:37.224645 5989 server_base.cc:1034] running on GCE node
W20260812 05:20:37.224601 5996 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 05:20:37.224609 5999 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 05:20:37.224962 5989 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 05:20:37.225172 5989 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 05:20:37.226307 5989 hybrid_clock.cc:648] HybridClock initialized: now 1786512037226297 us; error 26 us; skew 500 ppm
Aug 12 05:20:37 dist-test-slave-42z9 krb5kdc[5592](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512037, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.100.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 05:20:37.228622 5989 init.cc:377] Logged in from keytab as kudu/127.31.100.2@KRBTEST.COM (short username kudu)
I20260812 05:20:37.229027 5989 webserver.cc:460] Webserver started at http://127.31.100.2:44927/ using document root <none> and password file <none>
I20260812 05:20:37.229238 5989 fs_manager.cc:359] Metadata directory not provided
I20260812 05:20:37.229285 5989 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 05:20:37.229408 5989 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 05:20:37.230383 5989 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786512017613366-32144-0/minicluster-data/ts-1/data/instance:
uuid: "a827a8f22cd5406197a46f298dd5021b"
format_stamp: "Formatted at 2026-08-12 05:20:37 on dist-test-slave-42z9"
server_key: "30c8ec3c01b51058809baafa21b7aa69"
server_key_iv: "409045f923af9e817cb3aa9f4e79248d"
server_key_version: "encryptionkey@0"
I20260812 05:20:37.230706 5989 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786512017613366-32144-0/minicluster-data/ts-1/wal/instance:
uuid: "a827a8f22cd5406197a46f298dd5021b"
format_stamp: "Formatted at 2026-08-12 05:20:37 on dist-test-slave-42z9"
server_key: "30c8ec3c01b51058809baafa21b7aa69"
server_key_iv: "409045f923af9e817cb3aa9f4e79248d"
server_key_version: "encryptionkey@0"
I20260812 05:20:37.231956 5989 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260812 05:20:37.232595 6012 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 05:20:37.232744 5989 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260812 05:20:37.232818 5989 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786512017613366-32144-0/minicluster-data/ts-1/data,/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786512017613366-32144-0/minicluster-data/ts-1/wal
uuid: "a827a8f22cd5406197a46f298dd5021b"
format_stamp: "Formatted at 2026-08-12 05:20:37 on dist-test-slave-42z9"
server_key: "30c8ec3c01b51058809baafa21b7aa69"
server_key_iv: "409045f923af9e817cb3aa9f4e79248d"
server_key_version: "encryptionkey@0"
I20260812 05:20:37.232877 5989 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786512017613366-32144-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786512017613366-32144-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786512017613366-32144-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 05:20:37.243148 5989 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 05:20:37.244787 5989 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 05:20:37.244917 5989 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 05:20:37.245122 5989 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 05:20:37.245414 5989 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 05:20:37.245522 5989 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 05:20:37.245563 5989 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 05:20:37.245584 5989 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 05:20:37.251034 5989 rpc_server.cc:307] RPC server started. Bound to: 127.31.100.2:34047
I20260812 05:20:37.251092 6214 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.100.2:34047 every 8 connection(s)
I20260812 05:20:37.251366 5989 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786512017613366-32144-0/minicluster-data/ts-1/data/info.pb
Aug 12 05:20:37 dist-test-slave-42z9 krb5kdc[5592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512037, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.100.2@KRBTEST.COM for kudu/127.31.100.62@KRBTEST.COM
I20260812 05:20:37.259229 32144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0waFvT/build/release/bin/kudu as pid 5989
I20260812 05:20:37.259341 32144 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786512017613366-32144-0/minicluster-data/ts-1/wal/instance
I20260812 05:20:37.259586 32144 external_mini_cluster.cc:1428] Setting key 1ae2c6162b9f3a72aab180d00b9d8043
I20260812 05:20:37.259943 5768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:37.252019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.100.2:39903 (local address 127.31.100.62:40701)
0812 05:20:37.252094 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0812 05:20:37.252096 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 05:20:37.252226 (+ 130us) server_negotiation.cc:407] Connection header received
0812 05:20:37.253125 (+ 899us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:20:37.253126 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:20:37.253137 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:20:37.253192 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:20:37.253786 (+ 594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:37.254285 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:37.255187 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:37.255368 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:37.255413 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:37.257292 (+ 1879us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 05:20:37.257294 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 05:20:37.257294 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 05:20:37.257302 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0812 05:20:37.258525 (+ 1223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:20:37.258901 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:20:37.258902 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:20:37.258902 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:20:37.258959 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:20:37.259248 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:20:37.259249 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:20:37.259250 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:20:37.259405 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 05:20:37.259474 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0812 05:20:37.259747 (+ 273us) server_negotiation.cc:299] Negotiation successful
0812 05:20:37.259843 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260812 05:20:37.260066 6219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:37.251976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.62:40701 (local address 127.31.100.2:39903)
0812 05:20:37.252124 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0812 05:20:37.252157 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0812 05:20:37.253024 (+ 867us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:20:37.253254 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:20:37.253256 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:20:37.253542 (+ 286us) client_negotiation.cc:189] Negotiated authn=SASL
0812 05:20:37.253725 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:20:37.253730 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:20:37.254345 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:20:37.254346 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:20:37.255125 (+ 779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:20:37.255127 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:20:37.255420 (+ 293us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:20:37.255421 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:20:37.255513 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:37.255934 (+ 421us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 05:20:37.255942 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0812 05:20:37.257211 (+ 1269us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 05:20:37.258597 (+ 1386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 05:20:37.258604 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 05:20:37.258605 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 05:20:37.258811 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 05:20:37.259072 (+ 261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 05:20:37.259073 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 05:20:37.259073 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 05:20:37.259183 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 05:20:37.259464 (+ 281us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 05:20:37.259469 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 05:20:37.259582 (+ 113us) client_negotiation.cc:769] Sending connection context
0812 05:20:37.259706 (+ 124us) client_negotiation.cc:240] Negotiation successful
0812 05:20:37.259861 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":70,"threads_started":1}
I20260812 05:20:37.260706 6216 heartbeater.cc:347] Connected to a master server at 127.31.100.62:40701
I20260812 05:20:37.260805 6216 heartbeater.cc:464] Registering TS with master...
I20260812 05:20:37.261001 6216 heartbeater.cc:511] Master 127.31.100.62:40701 requested a full tablet report, sending...
I20260812 05:20:37.261364 5669 ts_manager.cc:194] Registered new tserver with Master: a827a8f22cd5406197a46f298dd5021b (127.31.100.2:34047)
I20260812 05:20:37.261732 5669 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.100.2@KRBTEST.COM'} at 127.31.100.2:39903
WARNING: no policy specified for kudu/127.31.100.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.100.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.100.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.3.keytab.
Entry for principal kudu/127.31.100.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.3.keytab.
I20260812 05:20:37.283560 32144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0waFvT/build/release/bin/kudu
/tmp/dist-test-task0waFvT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786512017613366-32144-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786512017613366-32144-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786512017613366-32144-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786512017613366-32144-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.100.3:0
--local_ip_for_outbound_sockets=127.31.100.3
--webserver_interface=127.31.100.3
--webserver_port=0
--tserver_master_addrs=127.31.100.62:40701
--keytab_file=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.3.keytab
--principal=kudu/127.31.100.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.100.20:34225
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786512017613366-32144-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786512017613366-32144-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786512017613366-32144-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 05:20:37.363857 6223 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 05:20:37.364058 6223 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 05:20:37.364080 6223 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 05:20:37.365199 6223 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 05:20:37.365242 6223 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 05:20:37.365267 6223 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.100.3
I20260812 05:20:37.366823 6223 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.100.20:34225
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786512017613366-32144-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786512017613366-32144-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.3.keytab
--principal=kudu/127.31.100.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.100.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786512017613366-32144-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.100.3
--webserver_port=0
--tserver_master_addrs=127.31.100.62:40701
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6223
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.100.3
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786512017613366-32144-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 04:20:34 UTC on ccycloud.quasar-phqdqp.root.comops.site
build id 4581871
I20260812 05:20:37.367050 6223 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 05:20:37.367260 6223 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 05:20:37.371973 6230 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 05:20:37.372036 6233 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 05:20:37.371974 6231 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 05:20:37.372121 6223 server_base.cc:1034] running on GCE node
I20260812 05:20:37.372324 6223 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 05:20:37.372547 6223 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 05:20:37.373729 6223 hybrid_clock.cc:648] HybridClock initialized: now 1786512037373713 us; error 32 us; skew 500 ppm
Aug 12 05:20:37 dist-test-slave-42z9 krb5kdc[5592](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512037, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.100.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 05:20:37.376027 6223 init.cc:377] Logged in from keytab as kudu/127.31.100.3@KRBTEST.COM (short username kudu)
I20260812 05:20:37.376463 6223 webserver.cc:460] Webserver started at http://127.31.100.3:37575/ using document root <none> and password file <none>
I20260812 05:20:37.376672 6223 fs_manager.cc:359] Metadata directory not provided
I20260812 05:20:37.376719 6223 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 05:20:37.376844 6223 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 05:20:37.377861 6223 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786512017613366-32144-0/minicluster-data/ts-2/data/instance:
uuid: "03413caaadf144de8e6a13d3c1988bd7"
format_stamp: "Formatted at 2026-08-12 05:20:37 on dist-test-slave-42z9"
server_key: "a3fcd824d4e1047a38664ba921749a9a"
server_key_iv: "2d43a33d5f5b417a27783b7222c7d989"
server_key_version: "encryptionkey@0"
I20260812 05:20:37.378175 6223 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786512017613366-32144-0/minicluster-data/ts-2/wal/instance:
uuid: "03413caaadf144de8e6a13d3c1988bd7"
format_stamp: "Formatted at 2026-08-12 05:20:37 on dist-test-slave-42z9"
server_key: "a3fcd824d4e1047a38664ba921749a9a"
server_key_iv: "2d43a33d5f5b417a27783b7222c7d989"
server_key_version: "encryptionkey@0"
I20260812 05:20:37.379335 6223 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260812 05:20:37.379982 6243 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 05:20:37.380131 6223 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260812 05:20:37.380214 6223 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786512017613366-32144-0/minicluster-data/ts-2/data,/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786512017613366-32144-0/minicluster-data/ts-2/wal
uuid: "03413caaadf144de8e6a13d3c1988bd7"
format_stamp: "Formatted at 2026-08-12 05:20:37 on dist-test-slave-42z9"
server_key: "a3fcd824d4e1047a38664ba921749a9a"
server_key_iv: "2d43a33d5f5b417a27783b7222c7d989"
server_key_version: "encryptionkey@0"
I20260812 05:20:37.380268 6223 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786512017613366-32144-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786512017613366-32144-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786512017613366-32144-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 05:20:37.389858 6223 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 05:20:37.391480 6223 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 05:20:37.391607 6223 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 05:20:37.391798 6223 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 05:20:37.392077 6223 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 05:20:37.392112 6223 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 05:20:37.392138 6223 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 05:20:37.392155 6223 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 05:20:37.397487 6223 rpc_server.cc:307] RPC server started. Bound to: 127.31.100.3:36891
I20260812 05:20:37.397544 6432 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.100.3:36891 every 8 connection(s)
I20260812 05:20:37.397817 6223 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786512017613366-32144-0/minicluster-data/ts-2/data/info.pb
I20260812 05:20:37.397986 32144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0waFvT/build/release/bin/kudu as pid 6223
I20260812 05:20:37.398101 32144 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786512017613366-32144-0/minicluster-data/ts-2/wal/instance
I20260812 05:20:37.398294 32144 external_mini_cluster.cc:1428] Setting key 89d6f20efecb2e50124c61830b5eb0b0
Aug 12 05:20:37 dist-test-slave-42z9 krb5kdc[5592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512037, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.100.3@KRBTEST.COM for kudu/127.31.100.62@KRBTEST.COM
I20260812 05:20:37.406173 5768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:37.398493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.100.3:56023 (local address 127.31.100.62:40701)
0812 05:20:37.398540 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0812 05:20:37.398542 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 05:20:37.398840 (+ 298us) server_negotiation.cc:407] Connection header received
0812 05:20:37.399805 (+ 965us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:20:37.399806 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:20:37.399816 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:20:37.399850 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:20:37.400392 (+ 542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:37.400870 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:37.401891 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:37.402077 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:37.402124 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:37.404040 (+ 1916us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 05:20:37.404042 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 05:20:37.404042 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 05:20:37.404048 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0812 05:20:37.405112 (+ 1064us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:20:37.405484 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:20:37.405484 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:20:37.405485 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:20:37.405534 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:20:37.405727 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:20:37.405727 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:20:37.405727 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:20:37.405817 (+ 90us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 05:20:37.405860 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0812 05:20:37.406058 (+ 198us) server_negotiation.cc:299] Negotiation successful
0812 05:20:37.406113 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260812 05:20:37.406332 6435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:37.398519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.62:40701 (local address 127.31.100.3:56023)
0812 05:20:37.398743 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0812 05:20:37.398782 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0812 05:20:37.399710 (+ 928us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:20:37.399911 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:20:37.399919 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:20:37.400151 (+ 232us) client_negotiation.cc:189] Negotiated authn=SASL
0812 05:20:37.400330 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:20:37.400335 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:20:37.400934 (+ 599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:20:37.400934 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:20:37.401830 (+ 896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:20:37.401833 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:20:37.402129 (+ 296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:20:37.402130 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:20:37.402220 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:37.402664 (+ 444us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 05:20:37.402672 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0812 05:20:37.403949 (+ 1277us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 05:20:37.405177 (+ 1228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 05:20:37.405183 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 05:20:37.405184 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 05:20:37.405384 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 05:20:37.405583 (+ 199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 05:20:37.405584 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 05:20:37.405584 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 05:20:37.405682 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 05:20:37.405857 (+ 175us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 05:20:37.405860 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 05:20:37.405967 (+ 107us) client_negotiation.cc:769] Sending connection context
0812 05:20:37.406040 (+ 73us) client_negotiation.cc:240] Negotiation successful
0812 05:20:37.406150 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":85,"threads_started":1}
I20260812 05:20:37.406769 6433 heartbeater.cc:347] Connected to a master server at 127.31.100.62:40701
I20260812 05:20:37.406855 6433 heartbeater.cc:464] Registering TS with master...
I20260812 05:20:37.407009 6433 heartbeater.cc:511] Master 127.31.100.62:40701 requested a full tablet report, sending...
I20260812 05:20:37.407382 5669 ts_manager.cc:194] Registered new tserver with Master: 03413caaadf144de8e6a13d3c1988bd7 (127.31.100.3:36891)
I20260812 05:20:37.407747 5669 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.100.3@KRBTEST.COM'} at 127.31.100.3:56023
I20260812 05:20:37.409966 32144 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 12 05:20:37 dist-test-slave-42z9 krb5kdc[5592](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512037, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 12 05:20:37 dist-test-slave-42z9 krb5kdc[5592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512037, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.100.1@KRBTEST.COM
I20260812 05:20:37.432554 6441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:37.425580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42470 (local address 127.31.100.1:38345)
0812 05:20:37.425781 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0812 05:20:37.425786 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 05:20:37.425797 (+ 11us) server_negotiation.cc:407] Connection header received
0812 05:20:37.425822 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:20:37.425824 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:20:37.425840 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:20:37.425946 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:20:37.426482 (+ 536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:37.427366 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:37.428075 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:37.428372 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:37.428433 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:37.430367 (+ 1934us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 05:20:37.430374 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 05:20:37.430374 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 05:20:37.430381 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 05:20:37.431720 (+ 1339us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:20:37.431989 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:20:37.431991 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:20:37.431992 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:20:37.432041 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:20:37.432170 (+ 129us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:20:37.432171 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:20:37.432171 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:20:37.432275 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 05:20:37.432364 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0812 05:20:37.432419 (+ 55us) server_negotiation.cc:299] Negotiation successful
0812 05:20:37.432475 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":166,"thread_start_us":92,"threads_started":1}
Aug 12 05:20:37 dist-test-slave-42z9 krb5kdc[5592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512037, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.100.62@KRBTEST.COM
I20260812 05:20:37.439725 5768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:37.433652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42776 (local address 127.31.100.62:40701)
0812 05:20:37.433708 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0812 05:20:37.433709 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0812 05:20:37.433717 (+ 8us) server_negotiation.cc:407] Connection header received
0812 05:20:37.433756 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:20:37.433757 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:20:37.433765 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:20:37.433816 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:20:37.434218 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:37.434686 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:37.435330 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:37.435535 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:37.435587 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:37.437679 (+ 2092us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 05:20:37.437681 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 05:20:37.437681 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 05:20:37.437689 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0812 05:20:37.438849 (+ 1160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:20:37.439133 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:20:37.439134 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:20:37.439134 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:20:37.439178 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:20:37.439293 (+ 115us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:20:37.439293 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:20:37.439294 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:20:37.439428 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 05:20:37.439494 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0812 05:20:37.439586 (+ 92us) server_negotiation.cc:299] Negotiation successful
0812 05:20:37.439652 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
W20260812 05:20:37.440109 5669 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42776
I20260812 05:20:37.440418 32144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0waFvT/build/release/bin/kudu with pid 5777
I20260812 05:20:37.461889 32144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0waFvT/build/release/bin/kudu with pid 5989
I20260812 05:20:37.468160 32144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0waFvT/build/release/bin/kudu with pid 6223
I20260812 05:20:37.478749 32144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0waFvT/build/release/bin/kudu with pid 5611
2026-08-12T05:20:37Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (1969 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786512017613366-32144-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 05:20:37 dist-test-slave-42z9 krb5kdc[6450](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 05:20:37 dist-test-slave-42z9 krb5kdc[6450](info): set up 2 sockets
Aug 12 05:20:37 dist-test-slave-42z9 krb5kdc[6450](info): commencing operation
krb5kdc: starting...
W20260812 05:20:38.640318 32144 mini_kdc.cc:121] Time spent starting KDC: real 1.139s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786512017613366-32144-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786512017613366-32144-0/krb5kdc/test-user.keytab.
Aug 12 05:20:38 dist-test-slave-42z9 krb5kdc[6450](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512038, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T05:20:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T05:20:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.100.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.100.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.100.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.62.keytab.
Entry for principal kudu/127.31.100.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.62.keytab.
I20260812 05:20:38.752368 32144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0waFvT/build/release/bin/kudu
/tmp/dist-test-task0waFvT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786512017613366-32144-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786512017613366-32144-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786512017613366-32144-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786512017613366-32144-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.100.62:40229
--webserver_interface=127.31.100.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.62.keytab
--principal=kudu/127.31.100.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.100.20:37865
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.100.62:40229
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786512017613366-32144-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786512017613366-32144-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786512017613366-32144-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 05:20:38.831631 6473 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 05:20:38.831816 6473 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 05:20:38.831837 6473 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 05:20:38.832950 6473 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 05:20:38.832990 6473 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 05:20:38.833002 6473 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 05:20:38.833014 6473 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 05:20:38.833024 6473 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 05:20:38.834509 6473 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.100.20:37865
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786512017613366-32144-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786512017613366-32144-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.100.62:40229
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.62.keytab
--principal=kudu/127.31.100.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.100.62:40229
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786512017613366-32144-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.100.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6473
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786512017613366-32144-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 04:20:34 UTC on ccycloud.quasar-phqdqp.root.comops.site
build id 4581871
I20260812 05:20:38.834736 6473 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 05:20:38.834931 6473 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 05:20:38.839452 6482 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 05:20:38.839460 6481 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 05:20:38.839460 6484 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 05:20:38.839635 6473 server_base.cc:1034] running on GCE node
I20260812 05:20:38.839864 6473 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 05:20:38.840164 6473 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 05:20:38.841385 6473 hybrid_clock.cc:648] HybridClock initialized: now 1786512038841371 us; error 33 us; skew 500 ppm
Aug 12 05:20:38 dist-test-slave-42z9 krb5kdc[6450](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512038, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.100.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 05:20:38.843752 6473 init.cc:377] Logged in from keytab as kudu/127.31.100.62@KRBTEST.COM (short username kudu)
I20260812 05:20:38.844054 6473 fs_manager.cc:359] Metadata directory not provided
I20260812 05:20:38.844098 6473 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 05:20:38.844192 6473 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 05:20:38.845156 6473 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786512017613366-32144-0/minicluster-data/master-0/data/instance:
uuid: "ba12523741ba469dab326c19c5fff78b"
format_stamp: "Formatted at 2026-08-12 05:20:38 on dist-test-slave-42z9"
server_key: "ad6f51ee682db414a58aa2729f1eed16"
server_key_iv: "b6afe62a30317daa737845b10320f44d"
server_key_version: "encryptionkey@0"
I20260812 05:20:38.845480 6473 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786512017613366-32144-0/minicluster-data/master-0/wal/instance:
uuid: "ba12523741ba469dab326c19c5fff78b"
format_stamp: "Formatted at 2026-08-12 05:20:38 on dist-test-slave-42z9"
server_key: "ad6f51ee682db414a58aa2729f1eed16"
server_key_iv: "b6afe62a30317daa737845b10320f44d"
server_key_version: "encryptionkey@0"
I20260812 05:20:38.846639 6473 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260812 05:20:38.847273 6499 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 05:20:38.847419 6473 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260812 05:20:38.847497 6473 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786512017613366-32144-0/minicluster-data/master-0/data,/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786512017613366-32144-0/minicluster-data/master-0/wal
uuid: "ba12523741ba469dab326c19c5fff78b"
format_stamp: "Formatted at 2026-08-12 05:20:38 on dist-test-slave-42z9"
server_key: "ad6f51ee682db414a58aa2729f1eed16"
server_key_iv: "b6afe62a30317daa737845b10320f44d"
server_key_version: "encryptionkey@0"
I20260812 05:20:38.847563 6473 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786512017613366-32144-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786512017613366-32144-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786512017613366-32144-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 05:20:38.857487 6473 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 05:20:38.859428 6473 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 05:20:38.859555 6473 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 05:20:38.863348 6473 rpc_server.cc:307] RPC server started. Bound to: 127.31.100.62:40229
I20260812 05:20:38.863373 6587 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.100.62:40229 every 8 connection(s)
I20260812 05:20:38.863641 6473 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786512017613366-32144-0/minicluster-data/master-0/data/info.pb
I20260812 05:20:38.864602 6589 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 05:20:38.866868 32144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0waFvT/build/release/bin/kudu as pid 6473
I20260812 05:20:38.866963 6589 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ba12523741ba469dab326c19c5fff78b: Bootstrap starting.
I20260812 05:20:38.866986 32144 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786512017613366-32144-0/minicluster-data/master-0/wal/instance
I20260812 05:20:38.867164 32144 external_mini_cluster.cc:1428] Setting key 87457bc442079e3e8fa08858b534c73c
I20260812 05:20:38.867637 6589 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ba12523741ba469dab326c19c5fff78b: Neither blocks nor log segments found. Creating new log.
I20260812 05:20:38.867973 6589 log.cc:824] T 00000000000000000000000000000000 P ba12523741ba469dab326c19c5fff78b: Log is configured to *not* fsync() on all Append() calls
I20260812 05:20:38.868680 6589 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ba12523741ba469dab326c19c5fff78b: No bootstrap required, opened a new log
I20260812 05:20:38.870818 6589 raft_consensus.cc:348] T 00000000000000000000000000000000 P ba12523741ba469dab326c19c5fff78b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba12523741ba469dab326c19c5fff78b" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 40229 } }
I20260812 05:20:38.870990 6589 raft_consensus.cc:374] T 00000000000000000000000000000000 P ba12523741ba469dab326c19c5fff78b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 05:20:38.871021 6589 raft_consensus.cc:729] T 00000000000000000000000000000000 P ba12523741ba469dab326c19c5fff78b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba12523741ba469dab326c19c5fff78b, State: Initialized, Role: FOLLOWER
I20260812 05:20:38.871141 6589 consensus_queue.cc:260] T 00000000000000000000000000000000 P ba12523741ba469dab326c19c5fff78b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba12523741ba469dab326c19c5fff78b" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 40229 } }
I20260812 05:20:38.871224 6589 raft_consensus.cc:388] T 00000000000000000000000000000000 P ba12523741ba469dab326c19c5fff78b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 05:20:38.871248 6589 raft_consensus.cc:482] T 00000000000000000000000000000000 P ba12523741ba469dab326c19c5fff78b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 05:20:38.871292 6589 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ba12523741ba469dab326c19c5fff78b [term 0 FOLLOWER]: Advancing to term 1
I20260812 05:20:38.871845 6589 raft_consensus.cc:504] T 00000000000000000000000000000000 P ba12523741ba469dab326c19c5fff78b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba12523741ba469dab326c19c5fff78b" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 40229 } }
I20260812 05:20:38.871969 6589 leader_election.cc:302] T 00000000000000000000000000000000 P ba12523741ba469dab326c19c5fff78b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ba12523741ba469dab326c19c5fff78b; no voters:
I20260812 05:20:38.872126 6589 leader_election.cc:288] T 00000000000000000000000000000000 P ba12523741ba469dab326c19c5fff78b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 05:20:38.872151 6603 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ba12523741ba469dab326c19c5fff78b [term 1 FOLLOWER]: Leader election won for term 1
I20260812 05:20:38.872258 6603 raft_consensus.cc:686] T 00000000000000000000000000000000 P ba12523741ba469dab326c19c5fff78b [term 1 LEADER]: Becoming Leader. State: Replica: ba12523741ba469dab326c19c5fff78b, State: Running, Role: LEADER
I20260812 05:20:38.872336 6603 consensus_queue.cc:237] T 00000000000000000000000000000000 P ba12523741ba469dab326c19c5fff78b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba12523741ba469dab326c19c5fff78b" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 40229 } }
I20260812 05:20:38.872406 6589 sys_catalog.cc:564] T 00000000000000000000000000000000 P ba12523741ba469dab326c19c5fff78b [sys.catalog]: configured and running, proceeding with master startup.
I20260812 05:20:38.872709 6606 sys_catalog.cc:455] T 00000000000000000000000000000000 P ba12523741ba469dab326c19c5fff78b [sys.catalog]: SysCatalogTable state changed. Reason: New leader ba12523741ba469dab326c19c5fff78b. Latest consensus state: current_term: 1 leader_uuid: "ba12523741ba469dab326c19c5fff78b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba12523741ba469dab326c19c5fff78b" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 40229 } } }
I20260812 05:20:38.872710 6605 sys_catalog.cc:455] T 00000000000000000000000000000000 P ba12523741ba469dab326c19c5fff78b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ba12523741ba469dab326c19c5fff78b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba12523741ba469dab326c19c5fff78b" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 40229 } } }
I20260812 05:20:38.872808 6606 sys_catalog.cc:458] T 00000000000000000000000000000000 P ba12523741ba469dab326c19c5fff78b [sys.catalog]: This master's current role is: LEADER
I20260812 05:20:38.872824 6605 sys_catalog.cc:458] T 00000000000000000000000000000000 P ba12523741ba469dab326c19c5fff78b [sys.catalog]: This master's current role is: LEADER
I20260812 05:20:38.873044 6610 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 05:20:38.873579 6610 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Aug 12 05:20:38 dist-test-slave-42z9 krb5kdc[6450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512038, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.100.62@KRBTEST.COM
I20260812 05:20:38.875267 6610 catalog_manager.cc:1347] Generated new cluster ID: b945f96101214734bdb01c11a22b3f74
I20260812 05:20:38.875320 6610 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 05:20:38.877617 6593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:38.867679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55906 (local address 127.31.100.62:40229)
0812 05:20:38.867822 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0812 05:20:38.867830 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 05:20:38.867854 (+ 24us) server_negotiation.cc:407] Connection header received
0812 05:20:38.868284 (+ 430us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:20:38.868291 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:20:38.868336 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:20:38.868453 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:20:38.869164 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:38.870066 (+ 902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:38.871167 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:38.871642 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:38.871723 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:38.874615 (+ 2892us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 05:20:38.874619 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 05:20:38.874619 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 05:20:38.874631 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0812 05:20:38.876216 (+ 1585us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:20:38.876537 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:20:38.876539 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:20:38.876540 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:20:38.876623 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:20:38.876800 (+ 177us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:20:38.876801 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:20:38.876801 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:20:38.877028 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 05:20:38.877171 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0812 05:20:38.877271 (+ 100us) server_negotiation.cc:299] Negotiation successful
0812 05:20:38.877402 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96,"thread_start_us":69,"threads_started":1}
I20260812 05:20:38.894698 6610 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 05:20:38.895403 6610 catalog_manager.cc:1504] Loading token signing keys...
I20260812 05:20:38.905821 6610 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ba12523741ba469dab326c19c5fff78b: Generated new TSK 0
I20260812 05:20:38.906015 6610 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.31.100.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.100.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.100.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.1.keytab.
Entry for principal kudu/127.31.100.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.1.keytab.
I20260812 05:20:38.934100 32144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0waFvT/build/release/bin/kudu
/tmp/dist-test-task0waFvT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786512017613366-32144-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786512017613366-32144-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786512017613366-32144-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786512017613366-32144-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.100.1:0
--local_ip_for_outbound_sockets=127.31.100.1
--webserver_interface=127.31.100.1
--webserver_port=0
--tserver_master_addrs=127.31.100.62:40229
--keytab_file=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.1.keytab
--principal=kudu/127.31.100.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.100.20:37865
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786512017613366-32144-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786512017613366-32144-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786512017613366-32144-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 05:20:39.035337 6637 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 05:20:39.035532 6637 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 05:20:39.035554 6637 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 05:20:39.036617 6637 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 05:20:39.036649 6637 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 05:20:39.036674 6637 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.100.1
I20260812 05:20:39.038131 6637 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.100.20:37865
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786512017613366-32144-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786512017613366-32144-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.1.keytab
--principal=kudu/127.31.100.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.100.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786512017613366-32144-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.100.1
--webserver_port=0
--tserver_master_addrs=127.31.100.62:40229
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6637
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.100.1
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786512017613366-32144-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 04:20:34 UTC on ccycloud.quasar-phqdqp.root.comops.site
build id 4581871
I20260812 05:20:39.038306 6637 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 05:20:39.038489 6637 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 05:20:39.043243 6652 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 05:20:39.043231 6651 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 05:20:39.043231 6656 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 05:20:39.043527 6637 server_base.cc:1034] running on GCE node
I20260812 05:20:39.043710 6637 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 05:20:39.043891 6637 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 05:20:39.045043 6637 hybrid_clock.cc:648] HybridClock initialized: now 1786512039045029 us; error 26 us; skew 500 ppm
Aug 12 05:20:39 dist-test-slave-42z9 krb5kdc[6450](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512039, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.100.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 05:20:39.047418 6637 init.cc:377] Logged in from keytab as kudu/127.31.100.1@KRBTEST.COM (short username kudu)
I20260812 05:20:39.047708 6637 fs_manager.cc:359] Metadata directory not provided
I20260812 05:20:39.047747 6637 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 05:20:39.047839 6637 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 05:20:39.048803 6637 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786512017613366-32144-0/minicluster-data/ts-0/data/instance:
uuid: "4683f014f8fa42cba4777302a9b587b5"
format_stamp: "Formatted at 2026-08-12 05:20:39 on dist-test-slave-42z9"
server_key: "9e80240a99f5ace6cfc29709a1d0a393"
server_key_iv: "6d697bff752aa642ccbe9287808e01d2"
server_key_version: "encryptionkey@0"
I20260812 05:20:39.049094 6637 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786512017613366-32144-0/minicluster-data/ts-0/wal/instance:
uuid: "4683f014f8fa42cba4777302a9b587b5"
format_stamp: "Formatted at 2026-08-12 05:20:39 on dist-test-slave-42z9"
server_key: "9e80240a99f5ace6cfc29709a1d0a393"
server_key_iv: "6d697bff752aa642ccbe9287808e01d2"
server_key_version: "encryptionkey@0"
I20260812 05:20:39.050350 6637 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260812 05:20:39.051070 6666 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 05:20:39.051230 6637 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260812 05:20:39.051323 6637 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786512017613366-32144-0/minicluster-data/ts-0/data,/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786512017613366-32144-0/minicluster-data/ts-0/wal
uuid: "4683f014f8fa42cba4777302a9b587b5"
format_stamp: "Formatted at 2026-08-12 05:20:39 on dist-test-slave-42z9"
server_key: "9e80240a99f5ace6cfc29709a1d0a393"
server_key_iv: "6d697bff752aa642ccbe9287808e01d2"
server_key_version: "encryptionkey@0"
I20260812 05:20:39.051391 6637 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786512017613366-32144-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786512017613366-32144-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786512017613366-32144-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 05:20:39.072286 6637 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 05:20:39.074429 6637 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 05:20:39.074549 6637 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 05:20:39.074748 6637 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 05:20:39.075003 6637 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 05:20:39.075033 6637 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 05:20:39.075063 6637 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 05:20:39.075074 6637 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 05:20:39.080579 6637 rpc_server.cc:307] RPC server started. Bound to: 127.31.100.1:45607
I20260812 05:20:39.080626 6856 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.100.1:45607 every 8 connection(s)
I20260812 05:20:39.080866 6637 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786512017613366-32144-0/minicluster-data/ts-0/data/info.pb
Aug 12 05:20:39 dist-test-slave-42z9 krb5kdc[6450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512039, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.100.1@KRBTEST.COM for kudu/127.31.100.62@KRBTEST.COM
I20260812 05:20:39.089526 32144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0waFvT/build/release/bin/kudu as pid 6637
I20260812 05:20:39.089634 32144 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786512017613366-32144-0/minicluster-data/ts-0/wal/instance
I20260812 05:20:39.089661 6593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:39.081549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.100.1:52043 (local address 127.31.100.62:40229)
0812 05:20:39.081595 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0812 05:20:39.081597 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 05:20:39.081785 (+ 188us) server_negotiation.cc:407] Connection header received
0812 05:20:39.082279 (+ 494us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:20:39.082281 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:20:39.082293 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:20:39.082346 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:20:39.083264 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:39.083772 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:39.084731 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:39.084948 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:39.085006 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:39.087156 (+ 2150us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 05:20:39.087158 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 05:20:39.087158 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 05:20:39.087166 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0812 05:20:39.088341 (+ 1175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:20:39.088704 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:20:39.088705 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:20:39.088706 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:20:39.088762 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:20:39.088982 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:20:39.088983 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:20:39.088983 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:20:39.089103 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 05:20:39.089159 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0812 05:20:39.089463 (+ 304us) server_negotiation.cc:299] Negotiation successful
0812 05:20:39.089567 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260812 05:20:39.089789 6860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:39.081512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.62:40229 (local address 127.31.100.1:52043)
0812 05:20:39.081662 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0812 05:20:39.081701 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0812 05:20:39.082177 (+ 476us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:20:39.082414 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:20:39.082418 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:20:39.082680 (+ 262us) client_negotiation.cc:189] Negotiated authn=SASL
0812 05:20:39.083162 (+ 482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:20:39.083168 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:20:39.083855 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:20:39.083856 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:20:39.084642 (+ 786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:20:39.084645 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:20:39.085020 (+ 375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:20:39.085021 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:20:39.085143 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:39.085611 (+ 468us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 05:20:39.085619 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0812 05:20:39.087059 (+ 1440us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 05:20:39.088415 (+ 1356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 05:20:39.088422 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 05:20:39.088424 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 05:20:39.088638 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 05:20:39.088816 (+ 178us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 05:20:39.088817 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 05:20:39.088817 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 05:20:39.088926 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 05:20:39.089158 (+ 232us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 05:20:39.089165 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 05:20:39.089315 (+ 150us) client_negotiation.cc:769] Sending connection context
0812 05:20:39.089401 (+ 86us) client_negotiation.cc:240] Negotiation successful
0812 05:20:39.089574 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":74,"threads_started":1}
I20260812 05:20:39.089830 32144 external_mini_cluster.cc:1428] Setting key b4aa0e20b3df86cce5e8bd238bfa89b9
I20260812 05:20:39.090327 6857 heartbeater.cc:347] Connected to a master server at 127.31.100.62:40229
I20260812 05:20:39.090430 6857 heartbeater.cc:464] Registering TS with master...
I20260812 05:20:39.090596 6857 heartbeater.cc:511] Master 127.31.100.62:40229 requested a full tablet report, sending...
I20260812 05:20:39.091128 6523 ts_manager.cc:194] Registered new tserver with Master: 4683f014f8fa42cba4777302a9b587b5 (127.31.100.1:45607)
I20260812 05:20:39.091703 6523 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.100.1@KRBTEST.COM'} at 127.31.100.1:52043
WARNING: no policy specified for kudu/127.31.100.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.100.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.100.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.2.keytab.
Entry for principal kudu/127.31.100.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.2.keytab.
I20260812 05:20:39.114032 32144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0waFvT/build/release/bin/kudu
/tmp/dist-test-task0waFvT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786512017613366-32144-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786512017613366-32144-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786512017613366-32144-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786512017613366-32144-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.100.2:0
--local_ip_for_outbound_sockets=127.31.100.2
--webserver_interface=127.31.100.2
--webserver_port=0
--tserver_master_addrs=127.31.100.62:40229
--keytab_file=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.2.keytab
--principal=kudu/127.31.100.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.100.20:37865
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786512017613366-32144-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786512017613366-32144-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786512017613366-32144-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 05:20:39.195154 6867 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 05:20:39.195346 6867 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 05:20:39.195369 6867 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 05:20:39.196439 6867 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 05:20:39.196472 6867 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 05:20:39.196503 6867 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.100.2
I20260812 05:20:39.197937 6867 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.100.20:37865
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786512017613366-32144-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786512017613366-32144-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.2.keytab
--principal=kudu/127.31.100.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.100.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786512017613366-32144-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.100.2
--webserver_port=0
--tserver_master_addrs=127.31.100.62:40229
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6867
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.100.2
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786512017613366-32144-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 04:20:34 UTC on ccycloud.quasar-phqdqp.root.comops.site
build id 4581871
I20260812 05:20:39.198107 6867 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 05:20:39.198275 6867 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 05:20:39.202983 6878 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 05:20:39.203008 6875 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 05:20:39.202980 6876 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 05:20:39.203058 6867 server_base.cc:1034] running on GCE node
I20260812 05:20:39.203394 6867 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 05:20:39.203653 6867 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 05:20:39.204802 6867 hybrid_clock.cc:648] HybridClock initialized: now 1786512039204784 us; error 33 us; skew 500 ppm
Aug 12 05:20:39 dist-test-slave-42z9 krb5kdc[6450](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512039, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.100.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 05:20:39.207142 6867 init.cc:377] Logged in from keytab as kudu/127.31.100.2@KRBTEST.COM (short username kudu)
I20260812 05:20:39.207430 6867 fs_manager.cc:359] Metadata directory not provided
I20260812 05:20:39.207507 6867 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 05:20:39.207625 6867 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 05:20:39.208585 6867 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786512017613366-32144-0/minicluster-data/ts-1/data/instance:
uuid: "74919813d1c04f7290598f757b305f8b"
format_stamp: "Formatted at 2026-08-12 05:20:39 on dist-test-slave-42z9"
server_key: "d6410012be155a3f427a64899a74432d"
server_key_iv: "fa3e6041e109e5f8fbe8528cff7c440a"
server_key_version: "encryptionkey@0"
I20260812 05:20:39.208894 6867 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786512017613366-32144-0/minicluster-data/ts-1/wal/instance:
uuid: "74919813d1c04f7290598f757b305f8b"
format_stamp: "Formatted at 2026-08-12 05:20:39 on dist-test-slave-42z9"
server_key: "d6410012be155a3f427a64899a74432d"
server_key_iv: "fa3e6041e109e5f8fbe8528cff7c440a"
server_key_version: "encryptionkey@0"
I20260812 05:20:39.210109 6867 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260812 05:20:39.210769 6884 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 05:20:39.210923 6867 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260812 05:20:39.211009 6867 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786512017613366-32144-0/minicluster-data/ts-1/data,/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786512017613366-32144-0/minicluster-data/ts-1/wal
uuid: "74919813d1c04f7290598f757b305f8b"
format_stamp: "Formatted at 2026-08-12 05:20:39 on dist-test-slave-42z9"
server_key: "d6410012be155a3f427a64899a74432d"
server_key_iv: "fa3e6041e109e5f8fbe8528cff7c440a"
server_key_version: "encryptionkey@0"
I20260812 05:20:39.211063 6867 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786512017613366-32144-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786512017613366-32144-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786512017613366-32144-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 05:20:39.230436 6867 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 05:20:39.232107 6867 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 05:20:39.232237 6867 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 05:20:39.232450 6867 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 05:20:39.232733 6867 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 05:20:39.232765 6867 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 05:20:39.232792 6867 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 05:20:39.232810 6867 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 05:20:39.238184 6867 rpc_server.cc:307] RPC server started. Bound to: 127.31.100.2:34517
I20260812 05:20:39.238247 7069 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.100.2:34517 every 8 connection(s)
I20260812 05:20:39.238476 6867 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786512017613366-32144-0/minicluster-data/ts-1/data/info.pb
Aug 12 05:20:39 dist-test-slave-42z9 krb5kdc[6450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512039, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.100.2@KRBTEST.COM for kudu/127.31.100.62@KRBTEST.COM
I20260812 05:20:39.246797 6593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:39.239124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.100.2:45621 (local address 127.31.100.62:40229)
0812 05:20:39.239187 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0812 05:20:39.239188 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0812 05:20:39.239356 (+ 168us) server_negotiation.cc:407] Connection header received
0812 05:20:39.239817 (+ 461us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:20:39.239818 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:20:39.239828 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:20:39.239872 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:20:39.240749 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:39.241330 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:39.242350 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:39.242577 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:39.242641 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:39.244443 (+ 1802us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 05:20:39.244445 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 05:20:39.244445 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 05:20:39.244452 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 05:20:39.245723 (+ 1271us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:20:39.246066 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:20:39.246067 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:20:39.246067 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:20:39.246123 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:20:39.246313 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:20:39.246313 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:20:39.246314 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:20:39.246413 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 05:20:39.246457 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0812 05:20:39.246663 (+ 206us) server_negotiation.cc:299] Negotiation successful
0812 05:20:39.246735 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260812 05:20:39.246932 7073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:39.239094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.62:40229 (local address 127.31.100.2:45621)
0812 05:20:39.239239 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0812 05:20:39.239277 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0812 05:20:39.239735 (+ 458us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:20:39.239939 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:20:39.239943 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:20:39.240195 (+ 252us) client_negotiation.cc:189] Negotiated authn=SASL
0812 05:20:39.240667 (+ 472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:20:39.240672 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:20:39.241408 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:20:39.241409 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:20:39.242244 (+ 835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:20:39.242246 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:20:39.242649 (+ 403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:20:39.242650 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:20:39.242743 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:39.243112 (+ 369us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 05:20:39.243119 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0812 05:20:39.244362 (+ 1243us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 05:20:39.245794 (+ 1432us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 05:20:39.245800 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 05:20:39.245801 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 05:20:39.246004 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 05:20:39.246175 (+ 171us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 05:20:39.246176 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 05:20:39.246176 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 05:20:39.246269 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 05:20:39.246472 (+ 203us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 05:20:39.246475 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 05:20:39.246574 (+ 99us) client_negotiation.cc:769] Sending connection context
0812 05:20:39.246647 (+ 73us) client_negotiation.cc:240] Negotiation successful
0812 05:20:39.246755 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":76,"threads_started":1}
I20260812 05:20:39.247367 7071 heartbeater.cc:347] Connected to a master server at 127.31.100.62:40229
I20260812 05:20:39.247473 7071 heartbeater.cc:464] Registering TS with master...
I20260812 05:20:39.247602 7071 heartbeater.cc:511] Master 127.31.100.62:40229 requested a full tablet report, sending...
I20260812 05:20:39.247933 6523 ts_manager.cc:194] Registered new tserver with Master: 74919813d1c04f7290598f757b305f8b (127.31.100.2:34517)
I20260812 05:20:39.248286 6523 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.100.2@KRBTEST.COM'} at 127.31.100.2:45621
I20260812 05:20:39.248586 32144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0waFvT/build/release/bin/kudu as pid 6867
I20260812 05:20:39.248683 32144 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786512017613366-32144-0/minicluster-data/ts-1/wal/instance
I20260812 05:20:39.248835 32144 external_mini_cluster.cc:1428] Setting key fc6b2a38943f701568504ea3b05e6907
WARNING: no policy specified for kudu/127.31.100.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.100.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.100.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.3.keytab.
Entry for principal kudu/127.31.100.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.3.keytab.
I20260812 05:20:39.272735 32144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0waFvT/build/release/bin/kudu
/tmp/dist-test-task0waFvT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786512017613366-32144-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786512017613366-32144-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786512017613366-32144-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786512017613366-32144-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.100.3:0
--local_ip_for_outbound_sockets=127.31.100.3
--webserver_interface=127.31.100.3
--webserver_port=0
--tserver_master_addrs=127.31.100.62:40229
--keytab_file=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.3.keytab
--principal=kudu/127.31.100.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.100.20:37865
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786512017613366-32144-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786512017613366-32144-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786512017613366-32144-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 05:20:39.352126 7078 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 05:20:39.352319 7078 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 05:20:39.352341 7078 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 05:20:39.353433 7078 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 05:20:39.353498 7078 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 05:20:39.353533 7078 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.100.3
I20260812 05:20:39.355070 7078 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.100.20:37865
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786512017613366-32144-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786512017613366-32144-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.3.keytab
--principal=kudu/127.31.100.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.100.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786512017613366-32144-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.100.3
--webserver_port=0
--tserver_master_addrs=127.31.100.62:40229
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7078
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.100.3
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786512017613366-32144-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 04:20:34 UTC on ccycloud.quasar-phqdqp.root.comops.site
build id 4581871
I20260812 05:20:39.355263 7078 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 05:20:39.355440 7078 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 05:20:39.360105 7091 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 05:20:39.360126 7087 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 05:20:39.360262 7078 server_base.cc:1034] running on GCE node
W20260812 05:20:39.360131 7086 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 05:20:39.360566 7078 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 05:20:39.360778 7078 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 05:20:39.361918 7078 hybrid_clock.cc:648] HybridClock initialized: now 1786512039361905 us; error 28 us; skew 500 ppm
Aug 12 05:20:39 dist-test-slave-42z9 krb5kdc[6450](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512039, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.100.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 05:20:39.364295 7078 init.cc:377] Logged in from keytab as kudu/127.31.100.3@KRBTEST.COM (short username kudu)
I20260812 05:20:39.364598 7078 fs_manager.cc:359] Metadata directory not provided
I20260812 05:20:39.364647 7078 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 05:20:39.364768 7078 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 05:20:39.365820 7078 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786512017613366-32144-0/minicluster-data/ts-2/data/instance:
uuid: "c6feb84df14c48379dca0c9105b73751"
format_stamp: "Formatted at 2026-08-12 05:20:39 on dist-test-slave-42z9"
server_key: "f50ec93fa2deff38dd136a5519c06bac"
server_key_iv: "2b4531cd1693eca04ea88c1890e7f635"
server_key_version: "encryptionkey@0"
I20260812 05:20:39.366140 7078 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786512017613366-32144-0/minicluster-data/ts-2/wal/instance:
uuid: "c6feb84df14c48379dca0c9105b73751"
format_stamp: "Formatted at 2026-08-12 05:20:39 on dist-test-slave-42z9"
server_key: "f50ec93fa2deff38dd136a5519c06bac"
server_key_iv: "2b4531cd1693eca04ea88c1890e7f635"
server_key_version: "encryptionkey@0"
I20260812 05:20:39.367336 7078 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260812 05:20:39.367976 7100 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 05:20:39.368120 7078 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260812 05:20:39.368204 7078 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786512017613366-32144-0/minicluster-data/ts-2/data,/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786512017613366-32144-0/minicluster-data/ts-2/wal
uuid: "c6feb84df14c48379dca0c9105b73751"
format_stamp: "Formatted at 2026-08-12 05:20:39 on dist-test-slave-42z9"
server_key: "f50ec93fa2deff38dd136a5519c06bac"
server_key_iv: "2b4531cd1693eca04ea88c1890e7f635"
server_key_version: "encryptionkey@0"
I20260812 05:20:39.368259 7078 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786512017613366-32144-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786512017613366-32144-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786512017613366-32144-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 05:20:39.383709 7078 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 05:20:39.385355 7078 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 05:20:39.385505 7078 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 05:20:39.385715 7078 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 05:20:39.386003 7078 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 05:20:39.386036 7078 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 05:20:39.386063 7078 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 05:20:39.386086 7078 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 05:20:39.391423 7078 rpc_server.cc:307] RPC server started. Bound to: 127.31.100.3:37617
I20260812 05:20:39.391474 7271 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.100.3:37617 every 8 connection(s)
I20260812 05:20:39.391690 7078 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786512017613366-32144-0/minicluster-data/ts-2/data/info.pb
Aug 12 05:20:39 dist-test-slave-42z9 krb5kdc[6450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512039, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.100.3@KRBTEST.COM for kudu/127.31.100.62@KRBTEST.COM
I20260812 05:20:39.397481 32144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0waFvT/build/release/bin/kudu as pid 7078
I20260812 05:20:39.397574 32144 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786512017613366-32144-0/minicluster-data/ts-2/wal/instance
I20260812 05:20:39.397737 32144 external_mini_cluster.cc:1428] Setting key df24e31588f4d512f739407f33ea4186
I20260812 05:20:39.400247 6593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:39.392321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.100.3:54845 (local address 127.31.100.62:40229)
0812 05:20:39.392376 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0812 05:20:39.392378 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 05:20:39.392560 (+ 182us) server_negotiation.cc:407] Connection header received
0812 05:20:39.393043 (+ 483us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:20:39.393044 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:20:39.393055 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:20:39.393111 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:20:39.394057 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:39.394558 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:39.395467 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:39.395643 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:39.395685 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:39.397751 (+ 2066us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 05:20:39.397753 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 05:20:39.397753 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 05:20:39.397761 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0812 05:20:39.399043 (+ 1282us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:20:39.399395 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:20:39.399396 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:20:39.399397 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:20:39.399450 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:20:39.399667 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:20:39.399668 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:20:39.399668 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:20:39.399775 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 05:20:39.399829 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0812 05:20:39.400080 (+ 251us) server_negotiation.cc:299] Negotiation successful
0812 05:20:39.400159 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260812 05:20:39.400403 7277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:39.392279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.62:40229 (local address 127.31.100.3:54845)
0812 05:20:39.392443 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0812 05:20:39.392483 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0812 05:20:39.392939 (+ 456us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:20:39.393169 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:20:39.393172 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:20:39.393484 (+ 312us) client_negotiation.cc:189] Negotiated authn=SASL
0812 05:20:39.393977 (+ 493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:20:39.393983 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:20:39.394636 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:20:39.394637 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:20:39.395410 (+ 773us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:20:39.395412 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:20:39.395728 (+ 316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:20:39.395729 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:20:39.395838 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:39.396246 (+ 408us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 05:20:39.396253 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0812 05:20:39.397668 (+ 1415us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 05:20:39.399115 (+ 1447us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 05:20:39.399121 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 05:20:39.399123 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 05:20:39.399327 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 05:20:39.399502 (+ 175us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 05:20:39.399503 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 05:20:39.399503 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 05:20:39.399613 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 05:20:39.399829 (+ 216us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 05:20:39.399836 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 05:20:39.399961 (+ 125us) client_negotiation.cc:769] Sending connection context
0812 05:20:39.400049 (+ 88us) client_negotiation.cc:240] Negotiation successful
0812 05:20:39.400200 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":83,"threads_started":1}
I20260812 05:20:39.400808 7273 heartbeater.cc:347] Connected to a master server at 127.31.100.62:40229
I20260812 05:20:39.400888 7273 heartbeater.cc:464] Registering TS with master...
I20260812 05:20:39.401006 7273 heartbeater.cc:511] Master 127.31.100.62:40229 requested a full tablet report, sending...
I20260812 05:20:39.401327 6523 ts_manager.cc:194] Registered new tserver with Master: c6feb84df14c48379dca0c9105b73751 (127.31.100.3:37617)
I20260812 05:20:39.401718 6523 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.100.3@KRBTEST.COM'} at 127.31.100.3:54845
I20260812 05:20:39.409281 32144 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 05:20:39.415508 6593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:39.410496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55916 (local address 127.31.100.62:40229)
0812 05:20:39.410544 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0812 05:20:39.410545 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0812 05:20:39.410621 (+ 76us) server_negotiation.cc:407] Connection header received
0812 05:20:39.410661 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:20:39.410662 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:20:39.410670 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:20:39.410711 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:20:39.411187 (+ 476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:39.411637 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:39.412286 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:39.412451 (+ 165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:39.412489 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:39.413520 (+ 1031us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 05:20:39.413521 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 05:20:39.413521 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 05:20:39.413528 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 05:20:39.414690 (+ 1162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:20:39.414969 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:20:39.414969 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:20:39.414970 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:20:39.415025 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:20:39.415138 (+ 113us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:20:39.415139 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:20:39.415139 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:20:39.415260 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 05:20:39.415328 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0812 05:20:39.415396 (+ 68us) server_negotiation.cc:299] Negotiation successful
0812 05:20:39.415453 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260812 05:20:39.416756 6523 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55916:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 05:20:39.418330 6523 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 05:20:39.424220 7296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:39.420573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.2:34517 (local address 127.0.0.1:48062)
0812 05:20:39.420911 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0812 05:20:39.420919 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0812 05:20:39.420995 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:20:39.421165 (+ 170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:20:39.421166 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:20:39.421168 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 05:20:39.421221 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:20:39.421222 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:20:39.422141 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:20:39.422142 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:20:39.423082 (+ 940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:20:39.423084 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:20:39.423949 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:20:39.423950 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:20:39.424032 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:39.424045 (+ 13us) client_negotiation.cc:769] Sending connection context
0812 05:20:39.424104 (+ 59us) client_negotiation.cc:240] Negotiation successful
0812 05:20:39.424153 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":323,"thread_start_us":92,"threads_started":1}
I20260812 05:20:39.424245 7298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:39.420697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48062 (local address 127.31.100.2:34517)
0812 05:20:39.420872 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0812 05:20:39.420876 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 05:20:39.420970 (+ 94us) server_negotiation.cc:407] Connection header received
0812 05:20:39.421081 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:20:39.421083 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:20:39.421098 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:20:39.421164 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 05:20:39.421310 (+ 146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:39.422056 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:39.423176 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:39.423880 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:39.423939 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:39.423999 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0812 05:20:39.424125 (+ 126us) server_negotiation.cc:299] Negotiation successful
0812 05:20:39.424166 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":72,"threads_started":1}
I20260812 05:20:39.424461 7295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:39.420509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36824 (local address 127.31.100.1:45607)
0812 05:20:39.420668 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0812 05:20:39.420673 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 05:20:39.420888 (+ 215us) server_negotiation.cc:407] Connection header received
0812 05:20:39.420947 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:20:39.420950 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:20:39.420966 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:20:39.421039 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 05:20:39.421283 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:39.422405 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:39.423476 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:39.424014 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:39.424083 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:39.424135 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0812 05:20:39.424339 (+ 204us) server_negotiation.cc:299] Negotiation successful
0812 05:20:39.424381 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":76,"threads_started":1}
I20260812 05:20:39.424681 7293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:39.420394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.1:45607 (local address 127.0.0.1:36824)
0812 05:20:39.420686 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0812 05:20:39.420723 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0812 05:20:39.420797 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:20:39.421055 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:20:39.421059 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:20:39.421061 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 05:20:39.421124 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:20:39.421126 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:20:39.422524 (+ 1398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:20:39.422525 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:20:39.423397 (+ 872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:20:39.423400 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:20:39.424170 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:20:39.424171 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:20:39.424250 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:39.424254 (+ 4us) client_negotiation.cc:769] Sending connection context
0812 05:20:39.424569 (+ 315us) client_negotiation.cc:240] Negotiation successful
0812 05:20:39.424615 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":52,"threads_started":1}
I20260812 05:20:39.424912 7299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:39.420751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.3:37617 (local address 127.0.0.1:40430)
0812 05:20:39.421205 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0812 05:20:39.421214 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0812 05:20:39.421260 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:20:39.421483 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:20:39.421484 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:20:39.421486 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 05:20:39.421529 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:20:39.421531 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:20:39.422748 (+ 1217us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:20:39.422749 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:20:39.423782 (+ 1033us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:20:39.423785 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:20:39.424700 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:20:39.424701 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:20:39.424768 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:39.424772 (+ 4us) client_negotiation.cc:769] Sending connection context
0812 05:20:39.424814 (+ 42us) client_negotiation.cc:240] Negotiation successful
0812 05:20:39.424850 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":435,"thread_start_us":58,"threads_started":1}
I20260812 05:20:39.425025 6969 tablet_service.cc:1467] Processing CreateTablet for tablet 42eea788e81741ec927823e5b271912f (DEFAULT_TABLE table=test-table [id=150c8863563f405a96e4bebef1e83f4a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 05:20:39.425375 6969 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 42eea788e81741ec927823e5b271912f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 05:20:39.425315 6747 tablet_service.cc:1467] Processing CreateTablet for tablet 42eea788e81741ec927823e5b271912f (DEFAULT_TABLE table=test-table [id=150c8863563f405a96e4bebef1e83f4a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 05:20:39.425697 6747 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 42eea788e81741ec927823e5b271912f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 05:20:39.427932 7301 tablet_bootstrap.cc:492] T 42eea788e81741ec927823e5b271912f P 74919813d1c04f7290598f757b305f8b: Bootstrap starting.
I20260812 05:20:39.428054 7302 tablet_bootstrap.cc:492] T 42eea788e81741ec927823e5b271912f P 4683f014f8fa42cba4777302a9b587b5: Bootstrap starting.
I20260812 05:20:39.428390 7300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:39.420833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40430 (local address 127.31.100.3:37617)
0812 05:20:39.421011 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0812 05:20:39.421014 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 05:20:39.421310 (+ 296us) server_negotiation.cc:407] Connection header received
0812 05:20:39.421349 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:20:39.421351 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:20:39.421365 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:20:39.421601 (+ 236us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 05:20:39.421670 (+ 69us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:39.422650 (+ 980us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:39.423866 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:39.424553 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:39.424617 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:39.424668 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0812 05:20:39.428267 (+ 3599us) server_negotiation.cc:299] Negotiation successful
0812 05:20:39.428313 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":97,"threads_started":1}
I20260812 05:20:39.428660 7302 tablet_bootstrap.cc:654] T 42eea788e81741ec927823e5b271912f P 4683f014f8fa42cba4777302a9b587b5: Neither blocks nor log segments found. Creating new log.
I20260812 05:20:39.428660 7301 tablet_bootstrap.cc:654] T 42eea788e81741ec927823e5b271912f P 74919813d1c04f7290598f757b305f8b: Neither blocks nor log segments found. Creating new log.
I20260812 05:20:39.428949 7173 tablet_service.cc:1467] Processing CreateTablet for tablet 42eea788e81741ec927823e5b271912f (DEFAULT_TABLE table=test-table [id=150c8863563f405a96e4bebef1e83f4a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 05:20:39.429226 7173 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 42eea788e81741ec927823e5b271912f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 05:20:39.429255 7301 log.cc:824] T 42eea788e81741ec927823e5b271912f P 74919813d1c04f7290598f757b305f8b: Log is configured to *not* fsync() on all Append() calls
I20260812 05:20:39.429255 7302 log.cc:824] T 42eea788e81741ec927823e5b271912f P 4683f014f8fa42cba4777302a9b587b5: Log is configured to *not* fsync() on all Append() calls
I20260812 05:20:39.429961 7302 tablet_bootstrap.cc:492] T 42eea788e81741ec927823e5b271912f P 4683f014f8fa42cba4777302a9b587b5: No bootstrap required, opened a new log
I20260812 05:20:39.430048 7302 ts_tablet_manager.cc:1397] T 42eea788e81741ec927823e5b271912f P 4683f014f8fa42cba4777302a9b587b5: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260812 05:20:39.430235 7301 tablet_bootstrap.cc:492] T 42eea788e81741ec927823e5b271912f P 74919813d1c04f7290598f757b305f8b: No bootstrap required, opened a new log
I20260812 05:20:39.430301 7301 ts_tablet_manager.cc:1397] T 42eea788e81741ec927823e5b271912f P 74919813d1c04f7290598f757b305f8b: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260812 05:20:39.431951 7302 raft_consensus.cc:348] T 42eea788e81741ec927823e5b271912f P 4683f014f8fa42cba4777302a9b587b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4683f014f8fa42cba4777302a9b587b5" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 45607 } } peers { permanent_uuid: "74919813d1c04f7290598f757b305f8b" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 34517 } } peers { permanent_uuid: "c6feb84df14c48379dca0c9105b73751" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 37617 } }
I20260812 05:20:39.432132 7302 raft_consensus.cc:374] T 42eea788e81741ec927823e5b271912f P 4683f014f8fa42cba4777302a9b587b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 05:20:39.432181 7302 raft_consensus.cc:729] T 42eea788e81741ec927823e5b271912f P 4683f014f8fa42cba4777302a9b587b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4683f014f8fa42cba4777302a9b587b5, State: Initialized, Role: FOLLOWER
I20260812 05:20:39.432155 7301 raft_consensus.cc:348] T 42eea788e81741ec927823e5b271912f P 74919813d1c04f7290598f757b305f8b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4683f014f8fa42cba4777302a9b587b5" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 45607 } } peers { permanent_uuid: "74919813d1c04f7290598f757b305f8b" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 34517 } } peers { permanent_uuid: "c6feb84df14c48379dca0c9105b73751" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 37617 } }
I20260812 05:20:39.432296 7301 raft_consensus.cc:374] T 42eea788e81741ec927823e5b271912f P 74919813d1c04f7290598f757b305f8b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 05:20:39.432341 7301 raft_consensus.cc:729] T 42eea788e81741ec927823e5b271912f P 74919813d1c04f7290598f757b305f8b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74919813d1c04f7290598f757b305f8b, State: Initialized, Role: FOLLOWER
I20260812 05:20:39.432364 7302 consensus_queue.cc:260] T 42eea788e81741ec927823e5b271912f P 4683f014f8fa42cba4777302a9b587b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4683f014f8fa42cba4777302a9b587b5" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 45607 } } peers { permanent_uuid: "74919813d1c04f7290598f757b305f8b" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 34517 } } peers { permanent_uuid: "c6feb84df14c48379dca0c9105b73751" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 37617 } }
I20260812 05:20:39.432443 7301 consensus_queue.cc:260] T 42eea788e81741ec927823e5b271912f P 74919813d1c04f7290598f757b305f8b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4683f014f8fa42cba4777302a9b587b5" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 45607 } } peers { permanent_uuid: "74919813d1c04f7290598f757b305f8b" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 34517 } } peers { permanent_uuid: "c6feb84df14c48379dca0c9105b73751" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 37617 } }
I20260812 05:20:39.432638 6857 heartbeater.cc:503] Master 127.31.100.62:40229 was elected leader, sending a full tablet report...
I20260812 05:20:39.432694 7301 ts_tablet_manager.cc:1428] T 42eea788e81741ec927823e5b271912f P 74919813d1c04f7290598f757b305f8b: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260812 05:20:39.432672 7309 raft_consensus.cc:482] T 42eea788e81741ec927823e5b271912f P 4683f014f8fa42cba4777302a9b587b5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 05:20:39.432655 7302 ts_tablet_manager.cc:1428] T 42eea788e81741ec927823e5b271912f P 4683f014f8fa42cba4777302a9b587b5: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260812 05:20:39.432753 7309 raft_consensus.cc:504] T 42eea788e81741ec927823e5b271912f P 4683f014f8fa42cba4777302a9b587b5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4683f014f8fa42cba4777302a9b587b5" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 45607 } } peers { permanent_uuid: "74919813d1c04f7290598f757b305f8b" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 34517 } } peers { permanent_uuid: "c6feb84df14c48379dca0c9105b73751" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 37617 } }
I20260812 05:20:39.432792 7071 heartbeater.cc:503] Master 127.31.100.62:40229 was elected leader, sending a full tablet report...
I20260812 05:20:39.433128 7307 tablet_bootstrap.cc:492] T 42eea788e81741ec927823e5b271912f P c6feb84df14c48379dca0c9105b73751: Bootstrap starting.
I20260812 05:20:39.433835 7307 tablet_bootstrap.cc:654] T 42eea788e81741ec927823e5b271912f P c6feb84df14c48379dca0c9105b73751: Neither blocks nor log segments found. Creating new log.
I20260812 05:20:39.434378 7307 log.cc:824] T 42eea788e81741ec927823e5b271912f P c6feb84df14c48379dca0c9105b73751: Log is configured to *not* fsync() on all Append() calls
I20260812 05:20:39.434685 7309 leader_election.cc:288] T 42eea788e81741ec927823e5b271912f P 4683f014f8fa42cba4777302a9b587b5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 74919813d1c04f7290598f757b305f8b (127.31.100.2:34517), c6feb84df14c48379dca0c9105b73751 (127.31.100.3:37617)
I20260812 05:20:39.435959 7307 tablet_bootstrap.cc:492] T 42eea788e81741ec927823e5b271912f P c6feb84df14c48379dca0c9105b73751: No bootstrap required, opened a new log
I20260812 05:20:39.436025 7307 ts_tablet_manager.cc:1397] T 42eea788e81741ec927823e5b271912f P c6feb84df14c48379dca0c9105b73751: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260812 05:20:39.436407 6860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:39.433286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.2:34517 (local address 127.31.100.1:33491)
0812 05:20:39.433368 (+ 82us) negotiation.cc:107] Waiting for socket to connect
0812 05:20:39.433378 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0812 05:20:39.433416 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:20:39.433814 (+ 398us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:20:39.433815 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:20:39.433817 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 05:20:39.433868 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:20:39.433870 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:20:39.434680 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:20:39.434681 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:20:39.435558 (+ 877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:20:39.435560 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:20:39.436175 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:20:39.436177 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:20:39.436257 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:39.436261 (+ 4us) client_negotiation.cc:769] Sending connection context
0812 05:20:39.436308 (+ 47us) client_negotiation.cc:240] Negotiation successful
0812 05:20:39.436345 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":66}
I20260812 05:20:39.436407 7298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:39.433431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.100.1:33491 (local address 127.31.100.2:34517)
0812 05:20:39.433480 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0812 05:20:39.433482 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 05:20:39.433490 (+ 8us) server_negotiation.cc:407] Connection header received
0812 05:20:39.433510 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:20:39.433511 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:20:39.433518 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:20:39.433563 (+ 45us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 05:20:39.433955 (+ 392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:39.434605 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:39.435640 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:39.436109 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:39.436168 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:39.436175 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0812 05:20:39.436322 (+ 147us) server_negotiation.cc:299] Negotiation successful
0812 05:20:39.436354 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260812 05:20:39.436733 7000 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42eea788e81741ec927823e5b271912f" candidate_uuid: "4683f014f8fa42cba4777302a9b587b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74919813d1c04f7290598f757b305f8b" is_pre_election: true
I20260812 05:20:39.436847 7000 raft_consensus.cc:2454] T 42eea788e81741ec927823e5b271912f P 74919813d1c04f7290598f757b305f8b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4683f014f8fa42cba4777302a9b587b5 in term 0.
I20260812 05:20:39.437640 6670 leader_election.cc:302] T 42eea788e81741ec927823e5b271912f P 4683f014f8fa42cba4777302a9b587b5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4683f014f8fa42cba4777302a9b587b5, 74919813d1c04f7290598f757b305f8b; no voters:
I20260812 05:20:39.437777 7309 raft_consensus.cc:2781] T 42eea788e81741ec927823e5b271912f P 4683f014f8fa42cba4777302a9b587b5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 05:20:39.437834 7309 raft_consensus.cc:482] T 42eea788e81741ec927823e5b271912f P 4683f014f8fa42cba4777302a9b587b5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 05:20:39.437870 7309 raft_consensus.cc:3037] T 42eea788e81741ec927823e5b271912f P 4683f014f8fa42cba4777302a9b587b5 [term 0 FOLLOWER]: Advancing to term 1
I20260812 05:20:39.437891 7313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:39.433589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.3:37617 (local address 127.31.100.1:59887)
0812 05:20:39.434803 (+ 1214us) negotiation.cc:107] Waiting for socket to connect
0812 05:20:39.434812 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0812 05:20:39.434865 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:20:39.434994 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:20:39.434995 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:20:39.434996 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 05:20:39.435059 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:20:39.435060 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:20:39.435584 (+ 524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:20:39.435585 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:20:39.436838 (+ 1253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:20:39.436840 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:20:39.437620 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:20:39.437621 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:20:39.437674 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:39.437677 (+ 3us) client_negotiation.cc:769] Sending connection context
0812 05:20:39.437726 (+ 49us) client_negotiation.cc:240] Negotiation successful
0812 05:20:39.437760 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1199,"thread_start_us":77,"threads_started":1}
I20260812 05:20:39.437887 7300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:39.433637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.100.1:59887 (local address 127.31.100.3:37617)
0812 05:20:39.433675 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0812 05:20:39.433677 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 05:20:39.434856 (+ 1179us) server_negotiation.cc:407] Connection header received
0812 05:20:39.434926 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:20:39.434927 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:20:39.434934 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:20:39.434977 (+ 43us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 05:20:39.435116 (+ 139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:39.435527 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:39.436928 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:39.437498 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:39.437546 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:39.437553 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0812 05:20:39.437797 (+ 244us) server_negotiation.cc:299] Negotiation successful
0812 05:20:39.437838 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260812 05:20:39.438216 7206 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42eea788e81741ec927823e5b271912f" candidate_uuid: "4683f014f8fa42cba4777302a9b587b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6feb84df14c48379dca0c9105b73751" is_pre_election: true
I20260812 05:20:39.437885 7307 raft_consensus.cc:348] T 42eea788e81741ec927823e5b271912f P c6feb84df14c48379dca0c9105b73751 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4683f014f8fa42cba4777302a9b587b5" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 45607 } } peers { permanent_uuid: "74919813d1c04f7290598f757b305f8b" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 34517 } } peers { permanent_uuid: "c6feb84df14c48379dca0c9105b73751" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 37617 } }
I20260812 05:20:39.438371 7307 raft_consensus.cc:374] T 42eea788e81741ec927823e5b271912f P c6feb84df14c48379dca0c9105b73751 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 05:20:39.438436 7307 raft_consensus.cc:729] T 42eea788e81741ec927823e5b271912f P c6feb84df14c48379dca0c9105b73751 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6feb84df14c48379dca0c9105b73751, State: Initialized, Role: FOLLOWER
I20260812 05:20:39.438556 7307 consensus_queue.cc:260] T 42eea788e81741ec927823e5b271912f P c6feb84df14c48379dca0c9105b73751 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4683f014f8fa42cba4777302a9b587b5" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 45607 } } peers { permanent_uuid: "74919813d1c04f7290598f757b305f8b" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 34517 } } peers { permanent_uuid: "c6feb84df14c48379dca0c9105b73751" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 37617 } }
I20260812 05:20:39.438629 7309 raft_consensus.cc:504] T 42eea788e81741ec927823e5b271912f P 4683f014f8fa42cba4777302a9b587b5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4683f014f8fa42cba4777302a9b587b5" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 45607 } } peers { permanent_uuid: "74919813d1c04f7290598f757b305f8b" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 34517 } } peers { permanent_uuid: "c6feb84df14c48379dca0c9105b73751" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 37617 } }
I20260812 05:20:39.438732 7206 raft_consensus.cc:2454] T 42eea788e81741ec927823e5b271912f P c6feb84df14c48379dca0c9105b73751 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4683f014f8fa42cba4777302a9b587b5 in term 0.
I20260812 05:20:39.438843 7309 leader_election.cc:288] T 42eea788e81741ec927823e5b271912f P 4683f014f8fa42cba4777302a9b587b5 [CANDIDATE]: Term 1 election: Requested vote from peers 74919813d1c04f7290598f757b305f8b (127.31.100.2:34517), c6feb84df14c48379dca0c9105b73751 (127.31.100.3:37617)
I20260812 05:20:39.438843 7307 ts_tablet_manager.cc:1428] T 42eea788e81741ec927823e5b271912f P c6feb84df14c48379dca0c9105b73751: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260812 05:20:39.438859 7000 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42eea788e81741ec927823e5b271912f" candidate_uuid: "4683f014f8fa42cba4777302a9b587b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74919813d1c04f7290598f757b305f8b"
I20260812 05:20:39.438951 7273 heartbeater.cc:503] Master 127.31.100.62:40229 was elected leader, sending a full tablet report...
I20260812 05:20:39.438941 7000 raft_consensus.cc:3037] T 42eea788e81741ec927823e5b271912f P 74919813d1c04f7290598f757b305f8b [term 0 FOLLOWER]: Advancing to term 1
I20260812 05:20:39.439023 7206 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42eea788e81741ec927823e5b271912f" candidate_uuid: "4683f014f8fa42cba4777302a9b587b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6feb84df14c48379dca0c9105b73751"
I20260812 05:20:39.439077 7206 raft_consensus.cc:3037] T 42eea788e81741ec927823e5b271912f P c6feb84df14c48379dca0c9105b73751 [term 0 FOLLOWER]: Advancing to term 1
I20260812 05:20:39.439754 7000 raft_consensus.cc:2454] T 42eea788e81741ec927823e5b271912f P 74919813d1c04f7290598f757b305f8b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4683f014f8fa42cba4777302a9b587b5 in term 1.
I20260812 05:20:39.439860 7206 raft_consensus.cc:2454] T 42eea788e81741ec927823e5b271912f P c6feb84df14c48379dca0c9105b73751 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4683f014f8fa42cba4777302a9b587b5 in term 1.
I20260812 05:20:39.439890 6670 leader_election.cc:302] T 42eea788e81741ec927823e5b271912f P 4683f014f8fa42cba4777302a9b587b5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4683f014f8fa42cba4777302a9b587b5, 74919813d1c04f7290598f757b305f8b; no voters:
I20260812 05:20:39.439994 7309 raft_consensus.cc:2781] T 42eea788e81741ec927823e5b271912f P 4683f014f8fa42cba4777302a9b587b5 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 05:20:39.440052 7309 raft_consensus.cc:686] T 42eea788e81741ec927823e5b271912f P 4683f014f8fa42cba4777302a9b587b5 [term 1 LEADER]: Becoming Leader. State: Replica: 4683f014f8fa42cba4777302a9b587b5, State: Running, Role: LEADER
I20260812 05:20:39.440121 7309 consensus_queue.cc:237] T 42eea788e81741ec927823e5b271912f P 4683f014f8fa42cba4777302a9b587b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4683f014f8fa42cba4777302a9b587b5" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 45607 } } peers { permanent_uuid: "74919813d1c04f7290598f757b305f8b" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 34517 } } peers { permanent_uuid: "c6feb84df14c48379dca0c9105b73751" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 37617 } }
I20260812 05:20:39.440786 6522 catalog_manager.cc:5495] T 42eea788e81741ec927823e5b271912f P 4683f014f8fa42cba4777302a9b587b5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4683f014f8fa42cba4777302a9b587b5 (127.31.100.1). New cstate: current_term: 1 leader_uuid: "4683f014f8fa42cba4777302a9b587b5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4683f014f8fa42cba4777302a9b587b5" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 45607 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "74919813d1c04f7290598f757b305f8b" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 34517 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c6feb84df14c48379dca0c9105b73751" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 37617 } health_report { overall_health: UNKNOWN } } }
I20260812 05:20:39.451296 7295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:39.448926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36832 (local address 127.31.100.1:45607)
0812 05:20:39.449027 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0812 05:20:39.449028 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0812 05:20:39.449035 (+ 7us) server_negotiation.cc:407] Connection header received
0812 05:20:39.449063 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:20:39.449064 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:20:39.449081 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:20:39.449138 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 05:20:39.449263 (+ 125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:39.449777 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:39.450602 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:39.450792 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:39.450841 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:39.450999 (+ 158us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 05:20:39.451106 (+ 107us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 05:20:39.451171 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0812 05:20:39.451216 (+ 45us) server_negotiation.cc:299] Negotiation successful
0812 05:20:39.451235 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260812 05:20:39.452745 7000 raft_consensus.cc:1261] T 42eea788e81741ec927823e5b271912f P 74919813d1c04f7290598f757b305f8b [term 1 FOLLOWER]: Refusing update from remote peer 4683f014f8fa42cba4777302a9b587b5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 05:20:39.452751 7206 raft_consensus.cc:1261] T 42eea788e81741ec927823e5b271912f P c6feb84df14c48379dca0c9105b73751 [term 1 FOLLOWER]: Refusing update from remote peer 4683f014f8fa42cba4777302a9b587b5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 05:20:39.453032 7310 consensus_queue.cc:1035] T 42eea788e81741ec927823e5b271912f P 4683f014f8fa42cba4777302a9b587b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74919813d1c04f7290598f757b305f8b" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 34517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 05:20:39.453142 7309 consensus_queue.cc:1035] T 42eea788e81741ec927823e5b271912f P 4683f014f8fa42cba4777302a9b587b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6feb84df14c48379dca0c9105b73751" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 37617 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 05:20:39.455788 7321 mvcc.cc:205] Tried to move back new op lower bound from 7317553313596248064 to 7317553313547251712. Current Snapshot: MvccSnapshot[applied={T|T < 7317553313596248064}]
I20260812 05:20:39.456457 7325 mvcc.cc:205] Tried to move back new op lower bound from 7317553313596248064 to 7317553313547251712. Current Snapshot: MvccSnapshot[applied={T|T < 7317553313596248064}]
I20260812 05:20:39.457757 6523 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55916:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 05:20:39.457850 6523 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55916:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 05:20:39.458520 6523 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ba12523741ba469dab326c19c5fff78b: Sending DeleteTablet for 3 replicas of tablet 42eea788e81741ec927823e5b271912f
I20260812 05:20:39.458838 6969 tablet_service.cc:1514] Processing DeleteTablet for tablet 42eea788e81741ec927823e5b271912f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 05:20:39 UTC) from {username='kudu', principal='kudu/127.31.100.62@KRBTEST.COM'} at 127.0.0.1:48062
I20260812 05:20:39.458901 6747 tablet_service.cc:1514] Processing DeleteTablet for tablet 42eea788e81741ec927823e5b271912f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 05:20:39 UTC) from {username='kudu', principal='kudu/127.31.100.62@KRBTEST.COM'} at 127.0.0.1:36824
I20260812 05:20:39.459082 7334 tablet_replica.cc:330] T 42eea788e81741ec927823e5b271912f P 74919813d1c04f7290598f757b305f8b: stopping tablet replica
I20260812 05:20:39.459107 7173 tablet_service.cc:1514] Processing DeleteTablet for tablet 42eea788e81741ec927823e5b271912f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 05:20:39 UTC) from {username='kudu', principal='kudu/127.31.100.62@KRBTEST.COM'} at 127.0.0.1:40430
I20260812 05:20:39.459174 7334 raft_consensus.cc:2229] T 42eea788e81741ec927823e5b271912f P 74919813d1c04f7290598f757b305f8b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 05:20:39.459264 7334 raft_consensus.cc:2258] T 42eea788e81741ec927823e5b271912f P 74919813d1c04f7290598f757b305f8b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 05:20:39.459308 32144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0waFvT/build/release/bin/kudu with pid 6637
I20260812 05:20:39.466099 7334 ts_tablet_manager.cc:1905] T 42eea788e81741ec927823e5b271912f P 74919813d1c04f7290598f757b305f8b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 05:20:39.467442 7334 ts_tablet_manager.cc:1918] T 42eea788e81741ec927823e5b271912f P 74919813d1c04f7290598f757b305f8b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 05:20:39.467518 7334 log.cc:1196] T 42eea788e81741ec927823e5b271912f P 74919813d1c04f7290598f757b305f8b: Deleting WAL directory at /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786512017613366-32144-0/minicluster-data/ts-1/wal/wals/42eea788e81741ec927823e5b271912f
I20260812 05:20:39.467733 7334 ts_tablet_manager.cc:1939] T 42eea788e81741ec927823e5b271912f P 74919813d1c04f7290598f757b305f8b: Deleting consensus metadata
I20260812 05:20:39.468124 6504 catalog_manager.cc:4841] TS 74919813d1c04f7290598f757b305f8b (127.31.100.2:34517): tablet 42eea788e81741ec927823e5b271912f (table test-table [id=150c8863563f405a96e4bebef1e83f4a]) successfully deleted
I20260812 05:20:39.470028 7336 tablet_replica.cc:330] T 42eea788e81741ec927823e5b271912f P c6feb84df14c48379dca0c9105b73751: stopping tablet replica
I20260812 05:20:39.470198 7336 raft_consensus.cc:2229] T 42eea788e81741ec927823e5b271912f P c6feb84df14c48379dca0c9105b73751 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 05:20:39.470335 7336 raft_consensus.cc:2258] T 42eea788e81741ec927823e5b271912f P c6feb84df14c48379dca0c9105b73751 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 05:20:39.474206 7336 ts_tablet_manager.cc:1905] T 42eea788e81741ec927823e5b271912f P c6feb84df14c48379dca0c9105b73751: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 05:20:39.475648 7336 ts_tablet_manager.cc:1918] T 42eea788e81741ec927823e5b271912f P c6feb84df14c48379dca0c9105b73751: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 05:20:39.475720 7336 log.cc:1196] T 42eea788e81741ec927823e5b271912f P c6feb84df14c48379dca0c9105b73751: Deleting WAL directory at /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786512017613366-32144-0/minicluster-data/ts-2/wal/wals/42eea788e81741ec927823e5b271912f
I20260812 05:20:39.475939 7336 ts_tablet_manager.cc:1939] T 42eea788e81741ec927823e5b271912f P c6feb84df14c48379dca0c9105b73751: Deleting consensus metadata
I20260812 05:20:39.476325 6504 catalog_manager.cc:4841] TS c6feb84df14c48379dca0c9105b73751 (127.31.100.3:37617): tablet 42eea788e81741ec927823e5b271912f (table test-table [id=150c8863563f405a96e4bebef1e83f4a]) successfully deleted
W20260812 05:20:39.480202 6504 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.31.100.1:45607 (error 108)
I20260812 05:20:39.480381 32144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0waFvT/build/release/bin/kudu with pid 6867
I20260812 05:20:39.482295 7299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:39.482131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.1:45607 (local address 127.0.0.1:36838)
0812 05:20:39.482197 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0812 05:20:39.482255 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.100.1:45607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260812 05:20:39.485694 6504 catalog_manager.cc:4584] TS 4683f014f8fa42cba4777302a9b587b5 (127.31.100.1:45607): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 42eea788e81741ec927823e5b271912f: Network error: Client connection negotiation failed: client connection to 127.31.100.1:45607: connect: Connection refused (error 111)
I20260812 05:20:39.489037 32144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0waFvT/build/release/bin/kudu with pid 7078
I20260812 05:20:39.502537 32144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0waFvT/build/release/bin/kudu with pid 6473
2026-08-12T05:20:39Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2024 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-08-12T05:20:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T05:20:39Z Disabled control of system clock
I20260812 05:20:39.533293 32144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0waFvT/build/release/bin/kudu
/tmp/dist-test-task0waFvT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786512017613366-32144-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786512017613366-32144-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786512017613366-32144-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786512017613366-32144-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.100.62:43449
--webserver_interface=127.31.100.62
--webserver_port=0
--builtin_ntp_servers=127.31.100.20:45535
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.100.62:43449
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260812 05:20:39.612880 7345 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 05:20:39.613066 7345 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 05:20:39.613086 7345 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 05:20:39.614153 7345 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 05:20:39.614187 7345 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 05:20:39.614197 7345 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 05:20:39.614208 7345 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 05:20:39.614218 7345 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 05:20:39.615494 7345 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.100.20:45535
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786512017613366-32144-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786512017613366-32144-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.100.62:43449
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.100.62:43449
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786512017613366-32144-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.100.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7345
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786512017613366-32144-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 04:20:34 UTC on ccycloud.quasar-phqdqp.root.comops.site
build id 4581871
I20260812 05:20:39.615681 7345 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 05:20:39.615911 7345 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 05:20:39.620504 7351 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 05:20:39.620555 7354 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 05:20:39.620551 7350 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 05:20:39.620647 7345 server_base.cc:1034] running on GCE node
I20260812 05:20:39.620929 7345 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 05:20:39.621191 7345 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 05:20:39.622342 7345 hybrid_clock.cc:648] HybridClock initialized: now 1786512039622325 us; error 26 us; skew 500 ppm
I20260812 05:20:39.623510 7345 webserver.cc:460] Webserver started at http://127.31.100.62:44065/ using document root <none> and password file <none>
I20260812 05:20:39.623729 7345 fs_manager.cc:359] Metadata directory not provided
I20260812 05:20:39.623775 7345 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 05:20:39.623900 7345 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 05:20:39.624943 7345 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786512017613366-32144-0/minicluster-data/master-0/data/instance:
uuid: "8e3cc5ca38384163bc37c77b67cd2946"
format_stamp: "Formatted at 2026-08-12 05:20:39 on dist-test-slave-42z9"
server_key: "e2583a341e2635a4ae52482367f3a114"
server_key_iv: "0239d633b1e11369d704862f1596ca19"
server_key_version: "encryptionkey@0"
I20260812 05:20:39.625299 7345 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786512017613366-32144-0/minicluster-data/master-0/wal/instance:
uuid: "8e3cc5ca38384163bc37c77b67cd2946"
format_stamp: "Formatted at 2026-08-12 05:20:39 on dist-test-slave-42z9"
server_key: "e2583a341e2635a4ae52482367f3a114"
server_key_iv: "0239d633b1e11369d704862f1596ca19"
server_key_version: "encryptionkey@0"
I20260812 05:20:39.626581 7345 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260812 05:20:39.627238 7361 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 05:20:39.627406 7345 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260812 05:20:39.627491 7345 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786512017613366-32144-0/minicluster-data/master-0/data,/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786512017613366-32144-0/minicluster-data/master-0/wal
uuid: "8e3cc5ca38384163bc37c77b67cd2946"
format_stamp: "Formatted at 2026-08-12 05:20:39 on dist-test-slave-42z9"
server_key: "e2583a341e2635a4ae52482367f3a114"
server_key_iv: "0239d633b1e11369d704862f1596ca19"
server_key_version: "encryptionkey@0"
I20260812 05:20:39.627555 7345 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786512017613366-32144-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786512017613366-32144-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786512017613366-32144-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 05:20:39.631074 7345 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 05:20:39.631314 7345 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 05:20:39.631444 7345 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 05:20:39.635172 7345 rpc_server.cc:307] RPC server started. Bound to: 127.31.100.62:43449
I20260812 05:20:39.635206 7446 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.100.62:43449 every 8 connection(s)
I20260812 05:20:39.635502 7345 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786512017613366-32144-0/minicluster-data/master-0/data/info.pb
I20260812 05:20:39.636134 7447 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 05:20:39.637090 32144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0waFvT/build/release/bin/kudu as pid 7345
I20260812 05:20:39.637199 32144 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786512017613366-32144-0/minicluster-data/master-0/wal/instance
I20260812 05:20:39.637347 32144 external_mini_cluster.cc:1428] Setting key c872101e340c1f8e847862094dd98b3e
I20260812 05:20:39.639400 7447 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8e3cc5ca38384163bc37c77b67cd2946: Bootstrap starting.
I20260812 05:20:39.639801 7455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:39.637863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51628 (local address 127.31.100.62:43449)
0812 05:20:39.638079 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0812 05:20:39.638087 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 05:20:39.638120 (+ 33us) server_negotiation.cc:407] Connection header received
0812 05:20:39.638614 (+ 494us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:20:39.638623 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:20:39.638677 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:20:39.638817 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:20:39.639379 (+ 562us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 05:20:39.639382 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 05:20:39.639382 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 05:20:39.639386 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0812 05:20:39.639429 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 05:20:39.639443 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 05:20:39.639508 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0812 05:20:39.639573 (+ 65us) server_negotiation.cc:299] Negotiation successful
0812 05:20:39.639626 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":97,"threads_started":1}
I20260812 05:20:39.640108 7447 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8e3cc5ca38384163bc37c77b67cd2946: Neither blocks nor log segments found. Creating new log.
I20260812 05:20:39.640394 7447 log.cc:824] T 00000000000000000000000000000000 P 8e3cc5ca38384163bc37c77b67cd2946: Log is configured to *not* fsync() on all Append() calls
I20260812 05:20:39.641006 7447 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8e3cc5ca38384163bc37c77b67cd2946: No bootstrap required, opened a new log
I20260812 05:20:39.642421 7447 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8e3cc5ca38384163bc37c77b67cd2946 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e3cc5ca38384163bc37c77b67cd2946" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 43449 } }
I20260812 05:20:39.642553 7447 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8e3cc5ca38384163bc37c77b67cd2946 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 05:20:39.642575 7447 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8e3cc5ca38384163bc37c77b67cd2946 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e3cc5ca38384163bc37c77b67cd2946, State: Initialized, Role: FOLLOWER
I20260812 05:20:39.642686 7447 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8e3cc5ca38384163bc37c77b67cd2946 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e3cc5ca38384163bc37c77b67cd2946" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 43449 } }
I20260812 05:20:39.642755 7447 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8e3cc5ca38384163bc37c77b67cd2946 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 05:20:39.642786 7447 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8e3cc5ca38384163bc37c77b67cd2946 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 05:20:39.642830 7447 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8e3cc5ca38384163bc37c77b67cd2946 [term 0 FOLLOWER]: Advancing to term 1
I20260812 05:20:39.643299 7447 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8e3cc5ca38384163bc37c77b67cd2946 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e3cc5ca38384163bc37c77b67cd2946" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 43449 } }
I20260812 05:20:39.643388 7447 leader_election.cc:302] T 00000000000000000000000000000000 P 8e3cc5ca38384163bc37c77b67cd2946 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8e3cc5ca38384163bc37c77b67cd2946; no voters:
I20260812 05:20:39.643537 7447 leader_election.cc:288] T 00000000000000000000000000000000 P 8e3cc5ca38384163bc37c77b67cd2946 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 05:20:39.643774 7447 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8e3cc5ca38384163bc37c77b67cd2946 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 05:20:39.643822 7459 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8e3cc5ca38384163bc37c77b67cd2946 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 05:20:39.643985 7459 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8e3cc5ca38384163bc37c77b67cd2946 [term 1 LEADER]: Becoming Leader. State: Replica: 8e3cc5ca38384163bc37c77b67cd2946, State: Running, Role: LEADER
I20260812 05:20:39.644083 7459 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8e3cc5ca38384163bc37c77b67cd2946 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e3cc5ca38384163bc37c77b67cd2946" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 43449 } }
I20260812 05:20:39.644390 7461 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8e3cc5ca38384163bc37c77b67cd2946 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8e3cc5ca38384163bc37c77b67cd2946" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e3cc5ca38384163bc37c77b67cd2946" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 43449 } } }
I20260812 05:20:39.644470 7461 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8e3cc5ca38384163bc37c77b67cd2946 [sys.catalog]: This master's current role is: LEADER
I20260812 05:20:39.644627 7463 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8e3cc5ca38384163bc37c77b67cd2946 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8e3cc5ca38384163bc37c77b67cd2946. Latest consensus state: current_term: 1 leader_uuid: "8e3cc5ca38384163bc37c77b67cd2946" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e3cc5ca38384163bc37c77b67cd2946" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 43449 } } }
I20260812 05:20:39.644712 7463 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8e3cc5ca38384163bc37c77b67cd2946 [sys.catalog]: This master's current role is: LEADER
I20260812 05:20:39.644659 7474 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 05:20:39.645376 7474 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 05:20:39.646746 7474 catalog_manager.cc:1347] Generated new cluster ID: 9041ee2027b942e2b32327e161843921
I20260812 05:20:39.646798 7474 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 05:20:39.672847 7474 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 05:20:39.673039 7474 catalog_manager.cc:1504] Loading token signing keys...
I20260812 05:20:39.677886 7474 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8e3cc5ca38384163bc37c77b67cd2946: Generated new TSK 0
I20260812 05:20:39.678056 7474 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 05:20:39.685029 32144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0waFvT/build/release/bin/kudu
/tmp/dist-test-task0waFvT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786512017613366-32144-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786512017613366-32144-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786512017613366-32144-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786512017613366-32144-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.100.1:0
--local_ip_for_outbound_sockets=127.31.100.1
--webserver_interface=127.31.100.1
--webserver_port=0
--tserver_master_addrs=127.31.100.62:43449
--builtin_ntp_servers=127.31.100.20:45535
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260812 05:20:39.801664 7489 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 05:20:39.801928 7489 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 05:20:39.801972 7489 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 05:20:39.803202 7489 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 05:20:39.803243 7489 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 05:20:39.803268 7489 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.100.1
I20260812 05:20:39.804558 7489 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.100.20:45535
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786512017613366-32144-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786512017613366-32144-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.100.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786512017613366-32144-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.100.1
--webserver_port=0
--tserver_master_addrs=127.31.100.62:43449
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7489
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.100.1
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786512017613366-32144-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 04:20:34 UTC on ccycloud.quasar-phqdqp.root.comops.site
build id 4581871
I20260812 05:20:39.804749 7489 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 05:20:39.804997 7489 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 05:20:39.809679 7497 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 05:20:39.809705 7496 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 05:20:39.809715 7489 server_base.cc:1034] running on GCE node
W20260812 05:20:39.809700 7500 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 05:20:39.810052 7489 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 05:20:39.810287 7489 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 05:20:39.811450 7489 hybrid_clock.cc:648] HybridClock initialized: now 1786512039811436 us; error 28 us; skew 500 ppm
I20260812 05:20:39.812615 7489 webserver.cc:460] Webserver started at http://127.31.100.1:39235/ using document root <none> and password file <none>
I20260812 05:20:39.812847 7489 fs_manager.cc:359] Metadata directory not provided
I20260812 05:20:39.812901 7489 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 05:20:39.813027 7489 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 05:20:39.814059 7489 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786512017613366-32144-0/minicluster-data/ts-0/data/instance:
uuid: "12f79c3e09af4f7fa1528829a8b22244"
format_stamp: "Formatted at 2026-08-12 05:20:39 on dist-test-slave-42z9"
server_key: "1611346cbb9b4f8019eb5f6eeb0e7969"
server_key_iv: "cb53ab041fe63a6da99faf209c34fbd2"
server_key_version: "encryptionkey@0"
I20260812 05:20:39.814389 7489 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786512017613366-32144-0/minicluster-data/ts-0/wal/instance:
uuid: "12f79c3e09af4f7fa1528829a8b22244"
format_stamp: "Formatted at 2026-08-12 05:20:39 on dist-test-slave-42z9"
server_key: "1611346cbb9b4f8019eb5f6eeb0e7969"
server_key_iv: "cb53ab041fe63a6da99faf209c34fbd2"
server_key_version: "encryptionkey@0"
I20260812 05:20:39.815613 7489 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260812 05:20:39.816249 7512 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 05:20:39.816421 7489 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260812 05:20:39.816509 7489 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786512017613366-32144-0/minicluster-data/ts-0/data,/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786512017613366-32144-0/minicluster-data/ts-0/wal
uuid: "12f79c3e09af4f7fa1528829a8b22244"
format_stamp: "Formatted at 2026-08-12 05:20:39 on dist-test-slave-42z9"
server_key: "1611346cbb9b4f8019eb5f6eeb0e7969"
server_key_iv: "cb53ab041fe63a6da99faf209c34fbd2"
server_key_version: "encryptionkey@0"
I20260812 05:20:39.816569 7489 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786512017613366-32144-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786512017613366-32144-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786512017613366-32144-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 05:20:39.819756 7489 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 05:20:39.819983 7489 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 05:20:39.820097 7489 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 05:20:39.820282 7489 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 05:20:39.820571 7489 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 05:20:39.820605 7489 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 05:20:39.820631 7489 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 05:20:39.820654 7489 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 05:20:39.826139 7489 rpc_server.cc:307] RPC server started. Bound to: 127.31.100.1:38135
I20260812 05:20:39.826195 7698 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.100.1:38135 every 8 connection(s)
I20260812 05:20:39.826472 7489 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786512017613366-32144-0/minicluster-data/ts-0/data/info.pb
I20260812 05:20:39.828881 7455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:39.827037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.100.1:55565 (local address 127.31.100.62:43449)
0812 05:20:39.827107 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0812 05:20:39.827109 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 05:20:39.827295 (+ 186us) server_negotiation.cc:407] Connection header received
0812 05:20:39.827685 (+ 390us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:20:39.827686 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:20:39.827692 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:20:39.827735 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:20:39.828705 (+ 970us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 05:20:39.828706 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 05:20:39.828706 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 05:20:39.828710 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0812 05:20:39.828724 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 05:20:39.828728 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 05:20:39.828769 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0812 05:20:39.828820 (+ 51us) server_negotiation.cc:299] Negotiation successful
0812 05:20:39.828830 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260812 05:20:39.829052 7703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:39.827025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.62:43449 (local address 127.31.100.1:55565)
0812 05:20:39.827194 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0812 05:20:39.827224 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0812 05:20:39.827616 (+ 392us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:20:39.827790 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:20:39.827793 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:20:39.827801 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0812 05:20:39.828578 (+ 777us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 05:20:39.828579 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0812 05:20:39.828604 (+ 25us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 05:20:39.828629 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 05:20:39.828772 (+ 143us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 05:20:39.828774 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 05:20:39.828776 (+ 2us) client_negotiation.cc:769] Sending connection context
0812 05:20:39.828820 (+ 44us) client_negotiation.cc:240] Negotiation successful
0812 05:20:39.828874 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":79,"threads_started":1}
I20260812 05:20:39.829548 7700 heartbeater.cc:347] Connected to a master server at 127.31.100.62:43449
I20260812 05:20:39.829629 7700 heartbeater.cc:464] Registering TS with master...
I20260812 05:20:39.829764 7700 heartbeater.cc:511] Master 127.31.100.62:43449 requested a full tablet report, sending...
I20260812 05:20:39.830196 7387 ts_manager.cc:194] Registered new tserver with Master: 12f79c3e09af4f7fa1528829a8b22244 (127.31.100.1:38135)
I20260812 05:20:39.830940 32144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0waFvT/build/release/bin/kudu as pid 7489
I20260812 05:20:39.831050 32144 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786512017613366-32144-0/minicluster-data/ts-0/wal/instance
I20260812 05:20:39.831207 32144 external_mini_cluster.cc:1428] Setting key 3c3b1e4691b165aa33c17544c1245343
I20260812 05:20:39.832020 32144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0waFvT/build/release/bin/kudu
/tmp/dist-test-task0waFvT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786512017613366-32144-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786512017613366-32144-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786512017613366-32144-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786512017613366-32144-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.100.2:0
--local_ip_for_outbound_sockets=127.31.100.2
--webserver_interface=127.31.100.2
--webserver_port=0
--tserver_master_addrs=127.31.100.62:43449
--builtin_ntp_servers=127.31.100.20:45535
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260812 05:20:39.911898 7706 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 05:20:39.912092 7706 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 05:20:39.912112 7706 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 05:20:39.913210 7706 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 05:20:39.913247 7706 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 05:20:39.913271 7706 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.100.2
I20260812 05:20:39.914597 7706 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.100.20:45535
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786512017613366-32144-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786512017613366-32144-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.100.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786512017613366-32144-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.100.2
--webserver_port=0
--tserver_master_addrs=127.31.100.62:43449
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7706
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.100.2
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786512017613366-32144-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 04:20:34 UTC on ccycloud.quasar-phqdqp.root.comops.site
build id 4581871
I20260812 05:20:39.914798 7706 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 05:20:39.915048 7706 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 05:20:39.919662 7712 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 05:20:39.919639 7718 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 05:20:39.919689 7713 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 05:20:39.919904 7706 server_base.cc:1034] running on GCE node
I20260812 05:20:39.920122 7706 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 05:20:39.920346 7706 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 05:20:39.921492 7706 hybrid_clock.cc:648] HybridClock initialized: now 1786512039921476 us; error 28 us; skew 500 ppm
I20260812 05:20:39.929859 7706 webserver.cc:460] Webserver started at http://127.31.100.2:34145/ using document root <none> and password file <none>
I20260812 05:20:39.930076 7706 fs_manager.cc:359] Metadata directory not provided
I20260812 05:20:39.930124 7706 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 05:20:39.930263 7706 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 05:20:39.931314 7706 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786512017613366-32144-0/minicluster-data/ts-1/data/instance:
uuid: "3ce57c0f4a284f6e85d6ab6c0889eae3"
format_stamp: "Formatted at 2026-08-12 05:20:39 on dist-test-slave-42z9"
server_key: "c5c3a7ec45986693fd9be18dd21c00a2"
server_key_iv: "f0cfd281c70074aa6156c47b7fe76146"
server_key_version: "encryptionkey@0"
I20260812 05:20:39.931651 7706 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786512017613366-32144-0/minicluster-data/ts-1/wal/instance:
uuid: "3ce57c0f4a284f6e85d6ab6c0889eae3"
format_stamp: "Formatted at 2026-08-12 05:20:39 on dist-test-slave-42z9"
server_key: "c5c3a7ec45986693fd9be18dd21c00a2"
server_key_iv: "f0cfd281c70074aa6156c47b7fe76146"
server_key_version: "encryptionkey@0"
I20260812 05:20:39.932857 7706 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260812 05:20:39.933562 7734 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 05:20:39.933709 7706 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260812 05:20:39.933792 7706 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786512017613366-32144-0/minicluster-data/ts-1/data,/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786512017613366-32144-0/minicluster-data/ts-1/wal
uuid: "3ce57c0f4a284f6e85d6ab6c0889eae3"
format_stamp: "Formatted at 2026-08-12 05:20:39 on dist-test-slave-42z9"
server_key: "c5c3a7ec45986693fd9be18dd21c00a2"
server_key_iv: "f0cfd281c70074aa6156c47b7fe76146"
server_key_version: "encryptionkey@0"
I20260812 05:20:39.933859 7706 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786512017613366-32144-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786512017613366-32144-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786512017613366-32144-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 05:20:39.937013 7706 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 05:20:39.937238 7706 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 05:20:39.937350 7706 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 05:20:39.937660 7706 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 05:20:39.937968 7706 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 05:20:39.938020 7706 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 05:20:39.938050 7706 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 05:20:39.938074 7706 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 05:20:39.943482 7706 rpc_server.cc:307] RPC server started. Bound to: 127.31.100.2:44059
I20260812 05:20:39.943547 7916 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.100.2:44059 every 8 connection(s)
I20260812 05:20:39.943832 7706 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786512017613366-32144-0/minicluster-data/ts-1/data/info.pb
I20260812 05:20:39.946530 7455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:39.944446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.100.2:43173 (local address 127.31.100.62:43449)
0812 05:20:39.944498 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0812 05:20:39.944500 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 05:20:39.944719 (+ 219us) server_negotiation.cc:407] Connection header received
0812 05:20:39.945186 (+ 467us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:20:39.945188 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:20:39.945195 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:20:39.945247 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:20:39.946288 (+ 1041us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 05:20:39.946289 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 05:20:39.946290 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 05:20:39.946294 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0812 05:20:39.946309 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 05:20:39.946314 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 05:20:39.946363 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0812 05:20:39.946438 (+ 75us) server_negotiation.cc:299] Negotiation successful
0812 05:20:39.946459 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260812 05:20:39.946708 7922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:39.944429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.62:43449 (local address 127.31.100.2:43173)
0812 05:20:39.944597 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0812 05:20:39.944625 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0812 05:20:39.945093 (+ 468us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:20:39.945309 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:20:39.945313 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:20:39.945322 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0812 05:20:39.946156 (+ 834us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 05:20:39.946156 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0812 05:20:39.946183 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 05:20:39.946207 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 05:20:39.946367 (+ 160us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 05:20:39.946371 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 05:20:39.946374 (+ 3us) client_negotiation.cc:769] Sending connection context
0812 05:20:39.946448 (+ 74us) client_negotiation.cc:240] Negotiation successful
0812 05:20:39.946507 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":75,"threads_started":1}
I20260812 05:20:39.946940 32144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0waFvT/build/release/bin/kudu as pid 7706
I20260812 05:20:39.947036 32144 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786512017613366-32144-0/minicluster-data/ts-1/wal/instance
I20260812 05:20:39.947120 7917 heartbeater.cc:347] Connected to a master server at 127.31.100.62:43449
I20260812 05:20:39.947208 7917 heartbeater.cc:464] Registering TS with master...
I20260812 05:20:39.947227 32144 external_mini_cluster.cc:1428] Setting key efe98dc66fb24cb9d7b1cba7f8362a88
I20260812 05:20:39.947355 7917 heartbeater.cc:511] Master 127.31.100.62:43449 requested a full tablet report, sending...
I20260812 05:20:39.947741 7386 ts_manager.cc:194] Registered new tserver with Master: 3ce57c0f4a284f6e85d6ab6c0889eae3 (127.31.100.2:44059)
I20260812 05:20:39.948127 32144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0waFvT/build/release/bin/kudu
/tmp/dist-test-task0waFvT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786512017613366-32144-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786512017613366-32144-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786512017613366-32144-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786512017613366-32144-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.100.3:0
--local_ip_for_outbound_sockets=127.31.100.3
--webserver_interface=127.31.100.3
--webserver_port=0
--tserver_master_addrs=127.31.100.62:43449
--builtin_ntp_servers=127.31.100.20:45535
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260812 05:20:40.026777 7923 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 05:20:40.026957 7923 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 05:20:40.026978 7923 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 05:20:40.028076 7923 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 05:20:40.028113 7923 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 05:20:40.028141 7923 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.100.3
I20260812 05:20:40.029419 7923 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.100.20:45535
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786512017613366-32144-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786512017613366-32144-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.100.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786512017613366-32144-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.100.3
--webserver_port=0
--tserver_master_addrs=127.31.100.62:43449
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7923
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.100.3
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786512017613366-32144-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 04:20:34 UTC on ccycloud.quasar-phqdqp.root.comops.site
build id 4581871
I20260812 05:20:40.029670 7923 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 05:20:40.029940 7923 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 05:20:40.034612 7931 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 05:20:40.034637 7936 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 05:20:40.034637 7928 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 05:20:40.034876 7923 server_base.cc:1034] running on GCE node
I20260812 05:20:40.035054 7923 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 05:20:40.035235 7923 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 05:20:40.036388 7923 hybrid_clock.cc:648] HybridClock initialized: now 1786512040036380 us; error 29 us; skew 500 ppm
I20260812 05:20:40.037549 7923 webserver.cc:460] Webserver started at http://127.31.100.3:45259/ using document root <none> and password file <none>
I20260812 05:20:40.037752 7923 fs_manager.cc:359] Metadata directory not provided
I20260812 05:20:40.037794 7923 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 05:20:40.037916 7923 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 05:20:40.038897 7923 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786512017613366-32144-0/minicluster-data/ts-2/data/instance:
uuid: "f41fee0ce94345348d210192a77ab24e"
format_stamp: "Formatted at 2026-08-12 05:20:40 on dist-test-slave-42z9"
server_key: "1d76656f91b7fdc239ed3bf778a100e1"
server_key_iv: "0d2e49eed7845a9cbd44963bea017bd7"
server_key_version: "encryptionkey@0"
I20260812 05:20:40.039214 7923 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786512017613366-32144-0/minicluster-data/ts-2/wal/instance:
uuid: "f41fee0ce94345348d210192a77ab24e"
format_stamp: "Formatted at 2026-08-12 05:20:40 on dist-test-slave-42z9"
server_key: "1d76656f91b7fdc239ed3bf778a100e1"
server_key_iv: "0d2e49eed7845a9cbd44963bea017bd7"
server_key_version: "encryptionkey@0"
I20260812 05:20:40.040371 7923 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260812 05:20:40.040970 7943 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 05:20:40.041106 7923 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260812 05:20:40.041188 7923 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786512017613366-32144-0/minicluster-data/ts-2/data,/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786512017613366-32144-0/minicluster-data/ts-2/wal
uuid: "f41fee0ce94345348d210192a77ab24e"
format_stamp: "Formatted at 2026-08-12 05:20:40 on dist-test-slave-42z9"
server_key: "1d76656f91b7fdc239ed3bf778a100e1"
server_key_iv: "0d2e49eed7845a9cbd44963bea017bd7"
server_key_version: "encryptionkey@0"
I20260812 05:20:40.041247 7923 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786512017613366-32144-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786512017613366-32144-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786512017613366-32144-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 05:20:40.044585 7923 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 05:20:40.044812 7923 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 05:20:40.044950 7923 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 05:20:40.045141 7923 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 05:20:40.045429 7923 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 05:20:40.045491 7923 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 05:20:40.045524 7923 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 05:20:40.045548 7923 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 05:20:40.050776 7923 rpc_server.cc:307] RPC server started. Bound to: 127.31.100.3:46643
I20260812 05:20:40.050832 8127 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.100.3:46643 every 8 connection(s)
I20260812 05:20:40.051110 7923 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786512017613366-32144-0/minicluster-data/ts-2/data/info.pb
I20260812 05:20:40.052873 32144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0waFvT/build/release/bin/kudu as pid 7923
I20260812 05:20:40.052977 32144 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786512017613366-32144-0/minicluster-data/ts-2/wal/instance
I20260812 05:20:40.053176 32144 external_mini_cluster.cc:1428] Setting key 375c4f45bb9dd7e813c711dd528b2acb
I20260812 05:20:40.053929 7455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:40.051728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.100.3:48019 (local address 127.31.100.62:43449)
0812 05:20:40.051778 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0812 05:20:40.051780 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 05:20:40.052005 (+ 225us) server_negotiation.cc:407] Connection header received
0812 05:20:40.052419 (+ 414us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:20:40.052420 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:20:40.052425 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:20:40.052516 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:20:40.053690 (+ 1174us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 05:20:40.053691 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 05:20:40.053691 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 05:20:40.053696 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0812 05:20:40.053713 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 05:20:40.053717 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 05:20:40.053763 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0812 05:20:40.053845 (+ 82us) server_negotiation.cc:299] Negotiation successful
0812 05:20:40.053860 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260812 05:20:40.054088 8133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:40.051714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.62:43449 (local address 127.31.100.3:48019)
0812 05:20:40.051888 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0812 05:20:40.051926 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0812 05:20:40.052345 (+ 419us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:20:40.052568 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:20:40.052571 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:20:40.052578 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0812 05:20:40.053554 (+ 976us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 05:20:40.053555 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0812 05:20:40.053583 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 05:20:40.053610 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 05:20:40.053765 (+ 155us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 05:20:40.053770 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 05:20:40.053773 (+ 3us) client_negotiation.cc:769] Sending connection context
0812 05:20:40.053844 (+ 71us) client_negotiation.cc:240] Negotiation successful
0812 05:20:40.053896 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":71,"threads_started":1}
I20260812 05:20:40.054438 8128 heartbeater.cc:347] Connected to a master server at 127.31.100.62:43449
I20260812 05:20:40.054522 8128 heartbeater.cc:464] Registering TS with master...
I20260812 05:20:40.054628 8128 heartbeater.cc:511] Master 127.31.100.62:43449 requested a full tablet report, sending...
I20260812 05:20:40.054903 7386 ts_manager.cc:194] Registered new tserver with Master: f41fee0ce94345348d210192a77ab24e (127.31.100.3:46643)
I20260812 05:20:40.064755 32144 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 05:20:40.066910 7455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:40.066053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34100 (local address 127.31.100.62:43449)
0812 05:20:40.066095 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0812 05:20:40.066096 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0812 05:20:40.066190 (+ 94us) server_negotiation.cc:407] Connection header received
0812 05:20:40.066242 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:20:40.066242 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:20:40.066248 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:20:40.066288 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:20:40.066748 (+ 460us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 05:20:40.066749 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 05:20:40.066749 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 05:20:40.066752 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0812 05:20:40.066764 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 05:20:40.066767 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 05:20:40.066805 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0812 05:20:40.066834 (+ 29us) server_negotiation.cc:299] Negotiation successful
0812 05:20:40.066844 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260812 05:20:40.068449 7386 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34100:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 05:20:40.069998 7386 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 05:20:40.073405 8140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:40.071833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.1:38135 (local address 127.0.0.1:58838)
0812 05:20:40.072144 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0812 05:20:40.072179 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0812 05:20:40.072227 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:20:40.072449 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:20:40.072449 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:20:40.072452 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0812 05:20:40.073011 (+ 559us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 05:20:40.073012 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0812 05:20:40.073016 (+ 4us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 05:20:40.073024 (+ 8us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 05:20:40.073252 (+ 228us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 05:20:40.073259 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 05:20:40.073262 (+ 3us) client_negotiation.cc:769] Sending connection context
0812 05:20:40.073327 (+ 65us) client_negotiation.cc:240] Negotiation successful
0812 05:20:40.073340 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":297,"thread_start_us":89,"threads_started":1}
I20260812 05:20:40.073467 8146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:40.071956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58838 (local address 127.31.100.1:38135)
0812 05:20:40.072329 (+ 373us) server_negotiation.cc:206] Beginning negotiation
0812 05:20:40.072332 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 05:20:40.072340 (+ 8us) server_negotiation.cc:407] Connection header received
0812 05:20:40.072360 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:20:40.072362 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:20:40.072369 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:20:40.072437 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:20:40.073097 (+ 660us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 05:20:40.073105 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 05:20:40.073105 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 05:20:40.073109 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0812 05:20:40.073146 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 05:20:40.073172 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 05:20:40.073225 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0812 05:20:40.073344 (+ 119us) server_negotiation.cc:299] Negotiation successful
0812 05:20:40.073361 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":342,"thread_start_us":110,"threads_started":1}
I20260812 05:20:40.073921 8143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:40.071819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.3:46643 (local address 127.0.0.1:40848)
0812 05:20:40.072132 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0812 05:20:40.072171 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0812 05:20:40.072223 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:20:40.073163 (+ 940us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:20:40.073164 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:20:40.073166 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0812 05:20:40.073577 (+ 411us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 05:20:40.073578 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0812 05:20:40.073585 (+ 7us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 05:20:40.073596 (+ 11us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 05:20:40.073789 (+ 193us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 05:20:40.073790 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 05:20:40.073790 (+ 0us) client_negotiation.cc:769] Sending connection context
0812 05:20:40.073838 (+ 48us) client_negotiation.cc:240] Negotiation successful
0812 05:20:40.073851 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":51,"threads_started":1}
I20260812 05:20:40.073948 8145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:40.071960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40848 (local address 127.31.100.3:46643)
0812 05:20:40.073047 (+ 1087us) server_negotiation.cc:206] Beginning negotiation
0812 05:20:40.073050 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 05:20:40.073060 (+ 10us) server_negotiation.cc:407] Connection header received
0812 05:20:40.073084 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:20:40.073086 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:20:40.073094 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:20:40.073167 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:20:40.073669 (+ 502us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 05:20:40.073676 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 05:20:40.073676 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 05:20:40.073680 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0812 05:20:40.073712 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 05:20:40.073734 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 05:20:40.073793 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0812 05:20:40.073846 (+ 53us) server_negotiation.cc:299] Negotiation successful
0812 05:20:40.073861 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1057,"thread_start_us":99,"threads_started":1}
I20260812 05:20:40.074062 8149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:40.072116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35434 (local address 127.31.100.2:44059)
0812 05:20:40.072306 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0812 05:20:40.072310 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 05:20:40.072319 (+ 9us) server_negotiation.cc:407] Connection header received
0812 05:20:40.072343 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:20:40.072345 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:20:40.072352 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:20:40.072409 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:20:40.073692 (+ 1283us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 05:20:40.073696 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 05:20:40.073696 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 05:20:40.073699 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0812 05:20:40.073727 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 05:20:40.073741 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 05:20:40.073798 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0812 05:20:40.073960 (+ 162us) server_negotiation.cc:299] Negotiation successful
0812 05:20:40.073977 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":159,"thread_start_us":83,"threads_started":1}
I20260812 05:20:40.073928 8148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:40.072011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.2:44059 (local address 127.0.0.1:35434)
0812 05:20:40.072155 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0812 05:20:40.072174 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0812 05:20:40.072223 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:20:40.072433 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:20:40.072445 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:20:40.072452 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0812 05:20:40.072972 (+ 520us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 05:20:40.072973 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0812 05:20:40.072988 (+ 15us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 05:20:40.073011 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 05:20:40.073821 (+ 810us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 05:20:40.073822 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 05:20:40.073822 (+ 0us) client_negotiation.cc:769] Sending connection context
0812 05:20:40.073862 (+ 40us) client_negotiation.cc:240] Negotiation successful
0812 05:20:40.073872 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":58,"threads_started":1}
I20260812 05:20:40.074440 7594 tablet_service.cc:1467] Processing CreateTablet for tablet d29e22570af84bbf9919265c96a47a53 (DEFAULT_TABLE table=test-table [id=ecfab01243134ebabc6e97b70964c37b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 05:20:40.074826 7594 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d29e22570af84bbf9919265c96a47a53. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 05:20:40.074802 7806 tablet_service.cc:1467] Processing CreateTablet for tablet d29e22570af84bbf9919265c96a47a53 (DEFAULT_TABLE table=test-table [id=ecfab01243134ebabc6e97b70964c37b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 05:20:40.075109 7806 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d29e22570af84bbf9919265c96a47a53. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 05:20:40.077425 8151 tablet_bootstrap.cc:492] T d29e22570af84bbf9919265c96a47a53 P 3ce57c0f4a284f6e85d6ab6c0889eae3: Bootstrap starting.
I20260812 05:20:40.077647 8150 tablet_bootstrap.cc:492] T d29e22570af84bbf9919265c96a47a53 P 12f79c3e09af4f7fa1528829a8b22244: Bootstrap starting.
I20260812 05:20:40.077812 8025 tablet_service.cc:1467] Processing CreateTablet for tablet d29e22570af84bbf9919265c96a47a53 (DEFAULT_TABLE table=test-table [id=ecfab01243134ebabc6e97b70964c37b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 05:20:40.078140 8025 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d29e22570af84bbf9919265c96a47a53. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 05:20:40.078168 8151 tablet_bootstrap.cc:654] T d29e22570af84bbf9919265c96a47a53 P 3ce57c0f4a284f6e85d6ab6c0889eae3: Neither blocks nor log segments found. Creating new log.
I20260812 05:20:40.078212 8150 tablet_bootstrap.cc:654] T d29e22570af84bbf9919265c96a47a53 P 12f79c3e09af4f7fa1528829a8b22244: Neither blocks nor log segments found. Creating new log.
I20260812 05:20:40.078466 8151 log.cc:824] T d29e22570af84bbf9919265c96a47a53 P 3ce57c0f4a284f6e85d6ab6c0889eae3: Log is configured to *not* fsync() on all Append() calls
I20260812 05:20:40.078495 8150 log.cc:824] T d29e22570af84bbf9919265c96a47a53 P 12f79c3e09af4f7fa1528829a8b22244: Log is configured to *not* fsync() on all Append() calls
I20260812 05:20:40.079170 8150 tablet_bootstrap.cc:492] T d29e22570af84bbf9919265c96a47a53 P 12f79c3e09af4f7fa1528829a8b22244: No bootstrap required, opened a new log
I20260812 05:20:40.079229 8150 ts_tablet_manager.cc:1397] T d29e22570af84bbf9919265c96a47a53 P 12f79c3e09af4f7fa1528829a8b22244: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260812 05:20:40.079429 8151 tablet_bootstrap.cc:492] T d29e22570af84bbf9919265c96a47a53 P 3ce57c0f4a284f6e85d6ab6c0889eae3: No bootstrap required, opened a new log
I20260812 05:20:40.079499 8151 ts_tablet_manager.cc:1397] T d29e22570af84bbf9919265c96a47a53 P 3ce57c0f4a284f6e85d6ab6c0889eae3: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260812 05:20:40.081127 8154 tablet_bootstrap.cc:492] T d29e22570af84bbf9919265c96a47a53 P f41fee0ce94345348d210192a77ab24e: Bootstrap starting.
I20260812 05:20:40.081367 8150 raft_consensus.cc:348] T d29e22570af84bbf9919265c96a47a53 P 12f79c3e09af4f7fa1528829a8b22244 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f41fee0ce94345348d210192a77ab24e" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 46643 } } peers { permanent_uuid: "3ce57c0f4a284f6e85d6ab6c0889eae3" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 44059 } } peers { permanent_uuid: "12f79c3e09af4f7fa1528829a8b22244" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 38135 } }
I20260812 05:20:40.081578 8150 raft_consensus.cc:374] T d29e22570af84bbf9919265c96a47a53 P 12f79c3e09af4f7fa1528829a8b22244 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 05:20:40.081624 8150 raft_consensus.cc:729] T d29e22570af84bbf9919265c96a47a53 P 12f79c3e09af4f7fa1528829a8b22244 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12f79c3e09af4f7fa1528829a8b22244, State: Initialized, Role: FOLLOWER
I20260812 05:20:40.081797 8154 tablet_bootstrap.cc:654] T d29e22570af84bbf9919265c96a47a53 P f41fee0ce94345348d210192a77ab24e: Neither blocks nor log segments found. Creating new log.
I20260812 05:20:40.081750 8150 consensus_queue.cc:260] T d29e22570af84bbf9919265c96a47a53 P 12f79c3e09af4f7fa1528829a8b22244 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f41fee0ce94345348d210192a77ab24e" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 46643 } } peers { permanent_uuid: "3ce57c0f4a284f6e85d6ab6c0889eae3" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 44059 } } peers { permanent_uuid: "12f79c3e09af4f7fa1528829a8b22244" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 38135 } }
I20260812 05:20:40.081871 8151 raft_consensus.cc:348] T d29e22570af84bbf9919265c96a47a53 P 3ce57c0f4a284f6e85d6ab6c0889eae3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f41fee0ce94345348d210192a77ab24e" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 46643 } } peers { permanent_uuid: "3ce57c0f4a284f6e85d6ab6c0889eae3" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 44059 } } peers { permanent_uuid: "12f79c3e09af4f7fa1528829a8b22244" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 38135 } }
I20260812 05:20:40.082024 8150 ts_tablet_manager.cc:1428] T d29e22570af84bbf9919265c96a47a53 P 12f79c3e09af4f7fa1528829a8b22244: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260812 05:20:40.082022 8151 raft_consensus.cc:374] T d29e22570af84bbf9919265c96a47a53 P 3ce57c0f4a284f6e85d6ab6c0889eae3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 05:20:40.082067 8151 raft_consensus.cc:729] T d29e22570af84bbf9919265c96a47a53 P 3ce57c0f4a284f6e85d6ab6c0889eae3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ce57c0f4a284f6e85d6ab6c0889eae3, State: Initialized, Role: FOLLOWER
I20260812 05:20:40.082106 8154 log.cc:824] T d29e22570af84bbf9919265c96a47a53 P f41fee0ce94345348d210192a77ab24e: Log is configured to *not* fsync() on all Append() calls
I20260812 05:20:40.082154 7700 heartbeater.cc:503] Master 127.31.100.62:43449 was elected leader, sending a full tablet report...
I20260812 05:20:40.082195 8151 consensus_queue.cc:260] T d29e22570af84bbf9919265c96a47a53 P 3ce57c0f4a284f6e85d6ab6c0889eae3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f41fee0ce94345348d210192a77ab24e" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 46643 } } peers { permanent_uuid: "3ce57c0f4a284f6e85d6ab6c0889eae3" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 44059 } } peers { permanent_uuid: "12f79c3e09af4f7fa1528829a8b22244" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 38135 } }
I20260812 05:20:40.082438 8151 ts_tablet_manager.cc:1428] T d29e22570af84bbf9919265c96a47a53 P 3ce57c0f4a284f6e85d6ab6c0889eae3: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260812 05:20:40.082556 7917 heartbeater.cc:503] Master 127.31.100.62:43449 was elected leader, sending a full tablet report...
I20260812 05:20:40.083067 8154 tablet_bootstrap.cc:492] T d29e22570af84bbf9919265c96a47a53 P f41fee0ce94345348d210192a77ab24e: No bootstrap required, opened a new log
I20260812 05:20:40.083140 8154 ts_tablet_manager.cc:1397] T d29e22570af84bbf9919265c96a47a53 P f41fee0ce94345348d210192a77ab24e: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260812 05:20:40.084964 8154 raft_consensus.cc:348] T d29e22570af84bbf9919265c96a47a53 P f41fee0ce94345348d210192a77ab24e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f41fee0ce94345348d210192a77ab24e" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 46643 } } peers { permanent_uuid: "3ce57c0f4a284f6e85d6ab6c0889eae3" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 44059 } } peers { permanent_uuid: "12f79c3e09af4f7fa1528829a8b22244" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 38135 } }
I20260812 05:20:40.085078 8154 raft_consensus.cc:374] T d29e22570af84bbf9919265c96a47a53 P f41fee0ce94345348d210192a77ab24e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 05:20:40.085098 8154 raft_consensus.cc:729] T d29e22570af84bbf9919265c96a47a53 P f41fee0ce94345348d210192a77ab24e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f41fee0ce94345348d210192a77ab24e, State: Initialized, Role: FOLLOWER
I20260812 05:20:40.085165 8154 consensus_queue.cc:260] T d29e22570af84bbf9919265c96a47a53 P f41fee0ce94345348d210192a77ab24e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f41fee0ce94345348d210192a77ab24e" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 46643 } } peers { permanent_uuid: "3ce57c0f4a284f6e85d6ab6c0889eae3" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 44059 } } peers { permanent_uuid: "12f79c3e09af4f7fa1528829a8b22244" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 38135 } }
I20260812 05:20:40.085345 8154 ts_tablet_manager.cc:1428] T d29e22570af84bbf9919265c96a47a53 P f41fee0ce94345348d210192a77ab24e: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260812 05:20:40.085388 8128 heartbeater.cc:503] Master 127.31.100.62:43449 was elected leader, sending a full tablet report...
W20260812 05:20:40.194392 7918 tablet.cc:2369] T d29e22570af84bbf9919265c96a47a53 P 3ce57c0f4a284f6e85d6ab6c0889eae3: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 05:20:40.301635 8129 tablet.cc:2369] T d29e22570af84bbf9919265c96a47a53 P f41fee0ce94345348d210192a77ab24e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 05:20:40.327152 7702 tablet.cc:2369] T d29e22570af84bbf9919265c96a47a53 P 12f79c3e09af4f7fa1528829a8b22244: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 05:20:40.366531 8157 raft_consensus.cc:482] T d29e22570af84bbf9919265c96a47a53 P 3ce57c0f4a284f6e85d6ab6c0889eae3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 05:20:40.366675 8157 raft_consensus.cc:504] T d29e22570af84bbf9919265c96a47a53 P 3ce57c0f4a284f6e85d6ab6c0889eae3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f41fee0ce94345348d210192a77ab24e" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 46643 } } peers { permanent_uuid: "3ce57c0f4a284f6e85d6ab6c0889eae3" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 44059 } } peers { permanent_uuid: "12f79c3e09af4f7fa1528829a8b22244" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 38135 } }
I20260812 05:20:40.367081 8157 leader_election.cc:288] T d29e22570af84bbf9919265c96a47a53 P 3ce57c0f4a284f6e85d6ab6c0889eae3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f41fee0ce94345348d210192a77ab24e (127.31.100.3:46643), 12f79c3e09af4f7fa1528829a8b22244 (127.31.100.1:38135)
I20260812 05:20:40.369486 7922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:40.368178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.3:46643 (local address 127.31.100.2:37675)
0812 05:20:40.368287 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0812 05:20:40.368298 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0812 05:20:40.368339 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:20:40.368637 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:20:40.368638 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:20:40.368641 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0812 05:20:40.369097 (+ 456us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 05:20:40.369097 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0812 05:20:40.369105 (+ 8us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 05:20:40.369118 (+ 13us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 05:20:40.369331 (+ 213us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 05:20:40.369332 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 05:20:40.369333 (+ 1us) client_negotiation.cc:769] Sending connection context
0812 05:20:40.369381 (+ 48us) client_negotiation.cc:240] Negotiation successful
0812 05:20:40.369395 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":91}
I20260812 05:20:40.369510 8145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:40.368437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.100.2:37675 (local address 127.31.100.3:46643)
0812 05:20:40.368479 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0812 05:20:40.368482 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 05:20:40.368489 (+ 7us) server_negotiation.cc:407] Connection header received
0812 05:20:40.368519 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:20:40.368520 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:20:40.368527 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:20:40.368575 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:20:40.369231 (+ 656us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 05:20:40.369232 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 05:20:40.369233 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 05:20:40.369238 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0812 05:20:40.369254 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 05:20:40.369258 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 05:20:40.369305 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0812 05:20:40.369396 (+ 91us) server_negotiation.cc:299] Negotiation successful
0812 05:20:40.369413 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260812 05:20:40.369741 8146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:40.368469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.100.2:34839 (local address 127.31.100.1:38135)
0812 05:20:40.368510 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0812 05:20:40.368512 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 05:20:40.368652 (+ 140us) server_negotiation.cc:407] Connection header received
0812 05:20:40.368710 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:20:40.368711 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:20:40.368716 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:20:40.368773 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:20:40.369327 (+ 554us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 05:20:40.369329 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 05:20:40.369329 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 05:20:40.369333 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0812 05:20:40.369348 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 05:20:40.369352 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 05:20:40.369401 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0812 05:20:40.369666 (+ 265us) server_negotiation.cc:299] Negotiation successful
0812 05:20:40.369682 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260812 05:20:40.369984 8057 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d29e22570af84bbf9919265c96a47a53" candidate_uuid: "3ce57c0f4a284f6e85d6ab6c0889eae3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f41fee0ce94345348d210192a77ab24e" is_pre_election: true
I20260812 05:20:40.370123 8057 raft_consensus.cc:2454] T d29e22570af84bbf9919265c96a47a53 P f41fee0ce94345348d210192a77ab24e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3ce57c0f4a284f6e85d6ab6c0889eae3 in term 0.
I20260812 05:20:40.370298 8160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:40.368256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.1:38135 (local address 127.31.100.2:34839)
0812 05:20:40.368568 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0812 05:20:40.368582 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0812 05:20:40.368658 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:20:40.368794 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:20:40.368795 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:20:40.368810 (+ 15us) client_negotiation.cc:189] Negotiated authn=SASL
0812 05:20:40.369211 (+ 401us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 05:20:40.369212 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0812 05:20:40.369220 (+ 8us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 05:20:40.369233 (+ 13us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 05:20:40.369475 (+ 242us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 05:20:40.369476 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 05:20:40.369477 (+ 1us) client_negotiation.cc:769] Sending connection context
0812 05:20:40.370191 (+ 714us) client_negotiation.cc:240] Negotiation successful
0812 05:20:40.370210 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":288,"thread_start_us":92,"threads_started":1}
I20260812 05:20:40.370404 7736 leader_election.cc:302] T d29e22570af84bbf9919265c96a47a53 P 3ce57c0f4a284f6e85d6ab6c0889eae3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3ce57c0f4a284f6e85d6ab6c0889eae3, f41fee0ce94345348d210192a77ab24e; no voters:
I20260812 05:20:40.370608 8157 raft_consensus.cc:2781] T d29e22570af84bbf9919265c96a47a53 P 3ce57c0f4a284f6e85d6ab6c0889eae3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 05:20:40.370632 8155 raft_consensus.cc:482] T d29e22570af84bbf9919265c96a47a53 P 12f79c3e09af4f7fa1528829a8b22244 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 05:20:40.370628 7626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d29e22570af84bbf9919265c96a47a53" candidate_uuid: "3ce57c0f4a284f6e85d6ab6c0889eae3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12f79c3e09af4f7fa1528829a8b22244" is_pre_election: true
I20260812 05:20:40.370743 8155 raft_consensus.cc:504] T d29e22570af84bbf9919265c96a47a53 P 12f79c3e09af4f7fa1528829a8b22244 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f41fee0ce94345348d210192a77ab24e" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 46643 } } peers { permanent_uuid: "3ce57c0f4a284f6e85d6ab6c0889eae3" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 44059 } } peers { permanent_uuid: "12f79c3e09af4f7fa1528829a8b22244" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 38135 } }
I20260812 05:20:40.370741 8157 raft_consensus.cc:482] T d29e22570af84bbf9919265c96a47a53 P 3ce57c0f4a284f6e85d6ab6c0889eae3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 05:20:40.370860 8157 raft_consensus.cc:3037] T d29e22570af84bbf9919265c96a47a53 P 3ce57c0f4a284f6e85d6ab6c0889eae3 [term 0 FOLLOWER]: Advancing to term 1
I20260812 05:20:40.371070 8155 leader_election.cc:288] T d29e22570af84bbf9919265c96a47a53 P 12f79c3e09af4f7fa1528829a8b22244 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f41fee0ce94345348d210192a77ab24e (127.31.100.3:46643), 3ce57c0f4a284f6e85d6ab6c0889eae3 (127.31.100.2:44059)
I20260812 05:20:40.371140 7626 raft_consensus.cc:2454] T d29e22570af84bbf9919265c96a47a53 P 12f79c3e09af4f7fa1528829a8b22244 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3ce57c0f4a284f6e85d6ab6c0889eae3 in term 0.
I20260812 05:20:40.371929 8157 raft_consensus.cc:504] T d29e22570af84bbf9919265c96a47a53 P 3ce57c0f4a284f6e85d6ab6c0889eae3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f41fee0ce94345348d210192a77ab24e" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 46643 } } peers { permanent_uuid: "3ce57c0f4a284f6e85d6ab6c0889eae3" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 44059 } } peers { permanent_uuid: "12f79c3e09af4f7fa1528829a8b22244" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 38135 } }
I20260812 05:20:40.372064 8157 leader_election.cc:288] T d29e22570af84bbf9919265c96a47a53 P 3ce57c0f4a284f6e85d6ab6c0889eae3 [CANDIDATE]: Term 1 election: Requested vote from peers f41fee0ce94345348d210192a77ab24e (127.31.100.3:46643), 12f79c3e09af4f7fa1528829a8b22244 (127.31.100.1:38135)
I20260812 05:20:40.372232 7626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d29e22570af84bbf9919265c96a47a53" candidate_uuid: "3ce57c0f4a284f6e85d6ab6c0889eae3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12f79c3e09af4f7fa1528829a8b22244"
I20260812 05:20:40.372314 7626 raft_consensus.cc:3037] T d29e22570af84bbf9919265c96a47a53 P 12f79c3e09af4f7fa1528829a8b22244 [term 0 FOLLOWER]: Advancing to term 1
I20260812 05:20:40.372378 8057 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d29e22570af84bbf9919265c96a47a53" candidate_uuid: "3ce57c0f4a284f6e85d6ab6c0889eae3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f41fee0ce94345348d210192a77ab24e"
I20260812 05:20:40.372476 8057 raft_consensus.cc:3037] T d29e22570af84bbf9919265c96a47a53 P f41fee0ce94345348d210192a77ab24e [term 0 FOLLOWER]: Advancing to term 1
I20260812 05:20:40.373173 8057 raft_consensus.cc:2454] T d29e22570af84bbf9919265c96a47a53 P f41fee0ce94345348d210192a77ab24e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3ce57c0f4a284f6e85d6ab6c0889eae3 in term 1.
I20260812 05:20:40.373211 7626 raft_consensus.cc:2454] T d29e22570af84bbf9919265c96a47a53 P 12f79c3e09af4f7fa1528829a8b22244 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3ce57c0f4a284f6e85d6ab6c0889eae3 in term 1.
I20260812 05:20:40.373383 7737 leader_election.cc:302] T d29e22570af84bbf9919265c96a47a53 P 3ce57c0f4a284f6e85d6ab6c0889eae3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12f79c3e09af4f7fa1528829a8b22244, 3ce57c0f4a284f6e85d6ab6c0889eae3; no voters:
I20260812 05:20:40.373525 8157 raft_consensus.cc:2781] T d29e22570af84bbf9919265c96a47a53 P 3ce57c0f4a284f6e85d6ab6c0889eae3 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 05:20:40.373658 8157 raft_consensus.cc:686] T d29e22570af84bbf9919265c96a47a53 P 3ce57c0f4a284f6e85d6ab6c0889eae3 [term 1 LEADER]: Becoming Leader. State: Replica: 3ce57c0f4a284f6e85d6ab6c0889eae3, State: Running, Role: LEADER
I20260812 05:20:40.373756 8157 consensus_queue.cc:237] T d29e22570af84bbf9919265c96a47a53 P 3ce57c0f4a284f6e85d6ab6c0889eae3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f41fee0ce94345348d210192a77ab24e" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 46643 } } peers { permanent_uuid: "3ce57c0f4a284f6e85d6ab6c0889eae3" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 44059 } } peers { permanent_uuid: "12f79c3e09af4f7fa1528829a8b22244" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 38135 } }
I20260812 05:20:40.374107 8149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:40.371675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.100.1:54911 (local address 127.31.100.2:44059)
0812 05:20:40.371783 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0812 05:20:40.371785 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 05:20:40.372259 (+ 474us) server_negotiation.cc:407] Connection header received
0812 05:20:40.372291 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:20:40.372292 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:20:40.372299 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:20:40.372437 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:20:40.373894 (+ 1457us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 05:20:40.373896 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 05:20:40.373896 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 05:20:40.373901 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0812 05:20:40.373921 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 05:20:40.373925 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 05:20:40.373954 (+ 29us) server_negotiation.cc:1035] Waiting for connection context
0812 05:20:40.374044 (+ 90us) server_negotiation.cc:299] Negotiation successful
0812 05:20:40.374055 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260812 05:20:40.374118 8163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:40.371331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.2:44059 (local address 127.31.100.1:54911)
0812 05:20:40.372184 (+ 853us) negotiation.cc:107] Waiting for socket to connect
0812 05:20:40.372194 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0812 05:20:40.372246 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:20:40.373338 (+ 1092us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:20:40.373339 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:20:40.373342 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0812 05:20:40.373774 (+ 432us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 05:20:40.373775 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0812 05:20:40.373781 (+ 6us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 05:20:40.373792 (+ 11us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 05:20:40.373994 (+ 202us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 05:20:40.373994 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 05:20:40.373995 (+ 1us) client_negotiation.cc:769] Sending connection context
0812 05:20:40.374031 (+ 36us) client_negotiation.cc:240] Negotiation successful
0812 05:20:40.374044 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":840,"thread_start_us":57,"threads_started":1}
I20260812 05:20:40.374428 7842 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d29e22570af84bbf9919265c96a47a53" candidate_uuid: "12f79c3e09af4f7fa1528829a8b22244" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ce57c0f4a284f6e85d6ab6c0889eae3" is_pre_election: true
I20260812 05:20:40.374605 7386 catalog_manager.cc:5495] T d29e22570af84bbf9919265c96a47a53 P 3ce57c0f4a284f6e85d6ab6c0889eae3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3ce57c0f4a284f6e85d6ab6c0889eae3 (127.31.100.2). New cstate: current_term: 1 leader_uuid: "3ce57c0f4a284f6e85d6ab6c0889eae3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f41fee0ce94345348d210192a77ab24e" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 46643 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3ce57c0f4a284f6e85d6ab6c0889eae3" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 44059 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "12f79c3e09af4f7fa1528829a8b22244" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 38135 } health_report { overall_health: UNKNOWN } } }
I20260812 05:20:40.374732 8145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:40.371466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.100.1:34869 (local address 127.31.100.3:46643)
0812 05:20:40.371500 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0812 05:20:40.371502 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 05:20:40.371509 (+ 7us) server_negotiation.cc:407] Connection header received
0812 05:20:40.371528 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:20:40.371528 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:20:40.371532 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:20:40.371585 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:20:40.374504 (+ 2919us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 05:20:40.374505 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 05:20:40.374506 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 05:20:40.374511 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0812 05:20:40.374527 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 05:20:40.374531 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 05:20:40.374561 (+ 30us) server_negotiation.cc:1035] Waiting for connection context
0812 05:20:40.374661 (+ 100us) server_negotiation.cc:299] Negotiation successful
0812 05:20:40.374673 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260812 05:20:40.374914 8165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:40.371202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.3:46643 (local address 127.31.100.1:34869)
0812 05:20:40.371424 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0812 05:20:40.371434 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0812 05:20:40.371476 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:20:40.371588 (+ 112us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:20:40.371588 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:20:40.371591 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0812 05:20:40.371980 (+ 389us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 05:20:40.371980 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0812 05:20:40.371988 (+ 8us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 05:20:40.372000 (+ 12us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 05:20:40.374604 (+ 2604us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 05:20:40.374604 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 05:20:40.374605 (+ 1us) client_negotiation.cc:769] Sending connection context
0812 05:20:40.374831 (+ 226us) client_negotiation.cc:240] Negotiation successful
0812 05:20:40.374844 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":50,"threads_started":1}
I20260812 05:20:40.375160 8057 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d29e22570af84bbf9919265c96a47a53" candidate_uuid: "12f79c3e09af4f7fa1528829a8b22244" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f41fee0ce94345348d210192a77ab24e" is_pre_election: true
I20260812 05:20:40.375243 8057 raft_consensus.cc:2379] T d29e22570af84bbf9919265c96a47a53 P f41fee0ce94345348d210192a77ab24e [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 12f79c3e09af4f7fa1528829a8b22244 in current term 1: Already voted for candidate 3ce57c0f4a284f6e85d6ab6c0889eae3 in this term.
I20260812 05:20:40.375523 7514 leader_election.cc:302] T d29e22570af84bbf9919265c96a47a53 P 12f79c3e09af4f7fa1528829a8b22244 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 12f79c3e09af4f7fa1528829a8b22244; no voters: 3ce57c0f4a284f6e85d6ab6c0889eae3, f41fee0ce94345348d210192a77ab24e
I20260812 05:20:40.375694 8155 raft_consensus.cc:2726] T d29e22570af84bbf9919265c96a47a53 P 12f79c3e09af4f7fa1528829a8b22244 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260812 05:20:40.425304 8149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:40.424276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35438 (local address 127.31.100.2:44059)
0812 05:20:40.424330 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0812 05:20:40.424331 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0812 05:20:40.424339 (+ 8us) server_negotiation.cc:407] Connection header received
0812 05:20:40.424403 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:20:40.424404 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:20:40.424418 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:20:40.424467 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:20:40.425012 (+ 545us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 05:20:40.425013 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 05:20:40.425014 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 05:20:40.425018 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0812 05:20:40.425033 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 05:20:40.425036 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 05:20:40.425082 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0812 05:20:40.425207 (+ 125us) server_negotiation.cc:299] Negotiation successful
0812 05:20:40.425227 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260812 05:20:40.427111 8057 raft_consensus.cc:1261] T d29e22570af84bbf9919265c96a47a53 P f41fee0ce94345348d210192a77ab24e [term 1 FOLLOWER]: Refusing update from remote peer 3ce57c0f4a284f6e85d6ab6c0889eae3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 05:20:40.427160 7626 raft_consensus.cc:1261] T d29e22570af84bbf9919265c96a47a53 P 12f79c3e09af4f7fa1528829a8b22244 [term 1 FOLLOWER]: Refusing update from remote peer 3ce57c0f4a284f6e85d6ab6c0889eae3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 05:20:40.427423 8157 consensus_queue.cc:1035] T d29e22570af84bbf9919265c96a47a53 P 3ce57c0f4a284f6e85d6ab6c0889eae3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f41fee0ce94345348d210192a77ab24e" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 46643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 05:20:40.427572 8166 consensus_queue.cc:1035] T d29e22570af84bbf9919265c96a47a53 P 3ce57c0f4a284f6e85d6ab6c0889eae3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "12f79c3e09af4f7fa1528829a8b22244" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 38135 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 05:20:40.430716 8172 mvcc.cc:205] Tried to move back new op lower bound from 7317553317586763776 to 7317553317371523072. Current Snapshot: MvccSnapshot[applied={T|T < 7317553317586763776}]
I20260812 05:20:40.431351 8173 mvcc.cc:205] Tried to move back new op lower bound from 7317553317586763776 to 7317553317371523072. Current Snapshot: MvccSnapshot[applied={T|T < 7317553317586763776}]
I20260812 05:20:40.433410 7386 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34100:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 05:20:40.433521 7386 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34100:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 05:20:40.434237 7386 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8e3cc5ca38384163bc37c77b67cd2946: Sending DeleteTablet for 3 replicas of tablet d29e22570af84bbf9919265c96a47a53
I20260812 05:20:40.434540 8025 tablet_service.cc:1514] Processing DeleteTablet for tablet d29e22570af84bbf9919265c96a47a53 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 05:20:40 UTC) from {username='slave'} at 127.0.0.1:40848
I20260812 05:20:40.434607 7806 tablet_service.cc:1514] Processing DeleteTablet for tablet d29e22570af84bbf9919265c96a47a53 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 05:20:40 UTC) from {username='slave'} at 127.0.0.1:35434
I20260812 05:20:40.434639 7594 tablet_service.cc:1514] Processing DeleteTablet for tablet d29e22570af84bbf9919265c96a47a53 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 05:20:40 UTC) from {username='slave'} at 127.0.0.1:58838
I20260812 05:20:40.434787 8186 tablet_replica.cc:330] T d29e22570af84bbf9919265c96a47a53 P 3ce57c0f4a284f6e85d6ab6c0889eae3: stopping tablet replica
I20260812 05:20:40.434888 8186 raft_consensus.cc:2229] T d29e22570af84bbf9919265c96a47a53 P 3ce57c0f4a284f6e85d6ab6c0889eae3 [term 1 LEADER]: Raft consensus shutting down.
I20260812 05:20:40.434885 8188 tablet_replica.cc:330] T d29e22570af84bbf9919265c96a47a53 P 12f79c3e09af4f7fa1528829a8b22244: stopping tablet replica
I20260812 05:20:40.434998 8188 raft_consensus.cc:2229] T d29e22570af84bbf9919265c96a47a53 P 12f79c3e09af4f7fa1528829a8b22244 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 05:20:40.435009 8186 raft_consensus.cc:2258] T d29e22570af84bbf9919265c96a47a53 P 3ce57c0f4a284f6e85d6ab6c0889eae3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 05:20:40.435075 32144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0waFvT/build/release/bin/kudu with pid 7489
I20260812 05:20:40.435091 8188 raft_consensus.cc:2258] T d29e22570af84bbf9919265c96a47a53 P 12f79c3e09af4f7fa1528829a8b22244 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 05:20:40.435582 8183 tablet_replica.cc:330] T d29e22570af84bbf9919265c96a47a53 P f41fee0ce94345348d210192a77ab24e: stopping tablet replica
I20260812 05:20:40.435663 8183 raft_consensus.cc:2229] T d29e22570af84bbf9919265c96a47a53 P f41fee0ce94345348d210192a77ab24e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 05:20:40.435736 8183 raft_consensus.cc:2258] T d29e22570af84bbf9919265c96a47a53 P f41fee0ce94345348d210192a77ab24e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 05:20:40.438251 8186 ts_tablet_manager.cc:1905] T d29e22570af84bbf9919265c96a47a53 P 3ce57c0f4a284f6e85d6ab6c0889eae3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 05:20:40.438279 8183 ts_tablet_manager.cc:1905] T d29e22570af84bbf9919265c96a47a53 P f41fee0ce94345348d210192a77ab24e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 05:20:40.439672 8183 ts_tablet_manager.cc:1918] T d29e22570af84bbf9919265c96a47a53 P f41fee0ce94345348d210192a77ab24e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 05:20:40.439728 8183 log.cc:1196] T d29e22570af84bbf9919265c96a47a53 P f41fee0ce94345348d210192a77ab24e: Deleting WAL directory at /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786512017613366-32144-0/minicluster-data/ts-2/wal/wals/d29e22570af84bbf9919265c96a47a53
I20260812 05:20:40.439916 8183 ts_tablet_manager.cc:1939] T d29e22570af84bbf9919265c96a47a53 P f41fee0ce94345348d210192a77ab24e: Deleting consensus metadata
I20260812 05:20:40.440116 8186 ts_tablet_manager.cc:1918] T d29e22570af84bbf9919265c96a47a53 P 3ce57c0f4a284f6e85d6ab6c0889eae3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 05:20:40.440188 8186 log.cc:1196] T d29e22570af84bbf9919265c96a47a53 P 3ce57c0f4a284f6e85d6ab6c0889eae3: Deleting WAL directory at /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786512017613366-32144-0/minicluster-data/ts-1/wal/wals/d29e22570af84bbf9919265c96a47a53
I20260812 05:20:40.440428 8186 ts_tablet_manager.cc:1939] T d29e22570af84bbf9919265c96a47a53 P 3ce57c0f4a284f6e85d6ab6c0889eae3: Deleting consensus metadata
I20260812 05:20:40.452795 7364 catalog_manager.cc:4841] TS 3ce57c0f4a284f6e85d6ab6c0889eae3 (127.31.100.2:44059): tablet d29e22570af84bbf9919265c96a47a53 (table test-table [id=ecfab01243134ebabc6e97b70964c37b]) successfully deleted
I20260812 05:20:40.452919 7364 catalog_manager.cc:4841] TS f41fee0ce94345348d210192a77ab24e (127.31.100.3:46643): tablet d29e22570af84bbf9919265c96a47a53 (table test-table [id=ecfab01243134ebabc6e97b70964c37b]) successfully deleted
W20260812 05:20:40.459236 7368 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.31.100.1:38135 (error 108)
I20260812 05:20:40.459801 8148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:40.459670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.1:38135 (local address 127.0.0.1:58840)
0812 05:20:40.459727 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0812 05:20:40.459770 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.100.1:38135: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260812 05:20:40.459918 7368 catalog_manager.cc:4584] TS 12f79c3e09af4f7fa1528829a8b22244 (127.31.100.1:38135): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d29e22570af84bbf9919265c96a47a53: Network error: Client connection negotiation failed: client connection to 127.31.100.1:38135: connect: Connection refused (error 111)
I20260812 05:20:40.460011 32144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0waFvT/build/release/bin/kudu with pid 7706
I20260812 05:20:40.470976 32144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0waFvT/build/release/bin/kudu with pid 7923
I20260812 05:20:40.481900 32144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0waFvT/build/release/bin/kudu with pid 7345
2026-08-12T05:20:40Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (980 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260812 05:20:40.507334 32144 mini_oidc.cc:152] Starting JWKS server
I20260812 05:20:40.508529 32144 webserver.cc:460] Webserver started at https://127.0.0.1:41091/ using document root <none> and password file <none>
I20260812 05:20:40.508626 32144 mini_oidc.cc:189] Starting OIDC Discovery server
I20260812 05:20:40.508800 32144 webserver.cc:460] Webserver started at http://127.0.0.1:34575/ using document root <none> and password file <none>
2026-08-12T05:20:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T05:20:40Z Disabled control of system clock
I20260812 05:20:40.517011 32144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0waFvT/build/release/bin/kudu
/tmp/dist-test-task0waFvT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786512017613366-32144-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786512017613366-32144-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786512017613366-32144-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786512017613366-32144-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.100.62:40801
--webserver_interface=127.31.100.62
--webserver_port=0
--builtin_ntp_servers=127.31.100.20:41549
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.100.62:40801
--enable_jwt_token_auth=true
--jwks_url=https://localhost:41091/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786512017613366-32144-0/testchainca.cert with env {}
W20260812 05:20:40.596745 8201 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 05:20:40.596942 8201 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 05:20:40.596968 8201 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 05:20:40.598094 8201 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 05:20:40.598129 8201 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 05:20:40.598140 8201 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 05:20:40.598151 8201 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 05:20:40.598162 8201 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260812 05:20:40.598173 8201 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260812 05:20:40.598183 8201 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:41091/jwks.json
I20260812 05:20:40.599684 8201 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.100.20:41549
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786512017613366-32144-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786512017613366-32144-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.100.62:40801
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.100.62:40801
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:41091/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786512017613366-32144-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.100.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786512017613366-32144-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8201
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786512017613366-32144-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 04:20:34 UTC on ccycloud.quasar-phqdqp.root.comops.site
build id 4581871
I20260812 05:20:40.599874 8201 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 05:20:40.600037 8201 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260812 05:20:40.604707 8201 server_base.cc:1034] running on GCE node
W20260812 05:20:40.604720 8208 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 05:20:40.604676 8207 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 05:20:40.604691 8210 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 05:20:40.605065 8201 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 05:20:40.605329 8201 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 05:20:40.606467 8201 hybrid_clock.cc:648] HybridClock initialized: now 1786512040606453 us; error 37 us; skew 500 ppm
I20260812 05:20:40.607583 8201 webserver.cc:460] Webserver started at http://127.31.100.62:35305/ using document root <none> and password file <none>
I20260812 05:20:40.607793 8201 fs_manager.cc:359] Metadata directory not provided
I20260812 05:20:40.607838 8201 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 05:20:40.607968 8201 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 05:20:40.608944 8201 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786512017613366-32144-0/minicluster-data/master-0/data/instance:
uuid: "3991b6671a674e5fa40e4e19e4ae98a5"
format_stamp: "Formatted at 2026-08-12 05:20:40 on dist-test-slave-42z9"
server_key: "20c73202f1d3e862e547865a7cea05dd"
server_key_iv: "6152ac64522d1940dc7814c96501ca8c"
server_key_version: "encryptionkey@0"
I20260812 05:20:40.609267 8201 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786512017613366-32144-0/minicluster-data/master-0/wal/instance:
uuid: "3991b6671a674e5fa40e4e19e4ae98a5"
format_stamp: "Formatted at 2026-08-12 05:20:40 on dist-test-slave-42z9"
server_key: "20c73202f1d3e862e547865a7cea05dd"
server_key_iv: "6152ac64522d1940dc7814c96501ca8c"
server_key_version: "encryptionkey@0"
I20260812 05:20:40.610544 8201 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260812 05:20:40.611268 8219 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 05:20:40.611415 8201 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260812 05:20:40.611502 8201 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786512017613366-32144-0/minicluster-data/master-0/data,/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786512017613366-32144-0/minicluster-data/master-0/wal
uuid: "3991b6671a674e5fa40e4e19e4ae98a5"
format_stamp: "Formatted at 2026-08-12 05:20:40 on dist-test-slave-42z9"
server_key: "20c73202f1d3e862e547865a7cea05dd"
server_key_iv: "6152ac64522d1940dc7814c96501ca8c"
server_key_version: "encryptionkey@0"
I20260812 05:20:40.611568 8201 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786512017613366-32144-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786512017613366-32144-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786512017613366-32144-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 05:20:40.620157 8201 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 05:20:40.620394 8201 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 05:20:40.620509 8201 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 05:20:40.624205 8201 rpc_server.cc:307] RPC server started. Bound to: 127.31.100.62:40801
I20260812 05:20:40.624238 8302 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.100.62:40801 every 8 connection(s)
I20260812 05:20:40.624516 8201 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786512017613366-32144-0/minicluster-data/master-0/data/info.pb
I20260812 05:20:40.625527 8303 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 05:20:40.627703 8303 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3991b6671a674e5fa40e4e19e4ae98a5: Bootstrap starting.
I20260812 05:20:40.628274 8303 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3991b6671a674e5fa40e4e19e4ae98a5: Neither blocks nor log segments found. Creating new log.
I20260812 05:20:40.628549 8303 log.cc:824] T 00000000000000000000000000000000 P 3991b6671a674e5fa40e4e19e4ae98a5: Log is configured to *not* fsync() on all Append() calls
I20260812 05:20:40.629253 8303 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3991b6671a674e5fa40e4e19e4ae98a5: No bootstrap required, opened a new log
I20260812 05:20:40.630851 8303 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3991b6671a674e5fa40e4e19e4ae98a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3991b6671a674e5fa40e4e19e4ae98a5" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 40801 } }
I20260812 05:20:40.630977 32144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0waFvT/build/release/bin/kudu as pid 8201
I20260812 05:20:40.630999 8303 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3991b6671a674e5fa40e4e19e4ae98a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 05:20:40.631024 8303 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3991b6671a674e5fa40e4e19e4ae98a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3991b6671a674e5fa40e4e19e4ae98a5, State: Initialized, Role: FOLLOWER
I20260812 05:20:40.631074 32144 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786512017613366-32144-0/minicluster-data/master-0/wal/instance
I20260812 05:20:40.631160 8303 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3991b6671a674e5fa40e4e19e4ae98a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3991b6671a674e5fa40e4e19e4ae98a5" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 40801 } }
I20260812 05:20:40.631227 8303 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3991b6671a674e5fa40e4e19e4ae98a5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 05:20:40.631258 8303 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3991b6671a674e5fa40e4e19e4ae98a5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 05:20:40.631271 32144 external_mini_cluster.cc:1428] Setting key 0aed1828dbf9c248cf6dac7056c02ff7
I20260812 05:20:40.631304 8303 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3991b6671a674e5fa40e4e19e4ae98a5 [term 0 FOLLOWER]: Advancing to term 1
I20260812 05:20:40.631968 8303 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3991b6671a674e5fa40e4e19e4ae98a5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3991b6671a674e5fa40e4e19e4ae98a5" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 40801 } }
I20260812 05:20:40.632081 8303 leader_election.cc:302] T 00000000000000000000000000000000 P 3991b6671a674e5fa40e4e19e4ae98a5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3991b6671a674e5fa40e4e19e4ae98a5; no voters:
I20260812 05:20:40.632226 8303 leader_election.cc:288] T 00000000000000000000000000000000 P 3991b6671a674e5fa40e4e19e4ae98a5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 05:20:40.632262 8312 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3991b6671a674e5fa40e4e19e4ae98a5 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 05:20:40.632390 8312 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3991b6671a674e5fa40e4e19e4ae98a5 [term 1 LEADER]: Becoming Leader. State: Replica: 3991b6671a674e5fa40e4e19e4ae98a5, State: Running, Role: LEADER
I20260812 05:20:40.632479 8303 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3991b6671a674e5fa40e4e19e4ae98a5 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 05:20:40.632499 8312 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3991b6671a674e5fa40e4e19e4ae98a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3991b6671a674e5fa40e4e19e4ae98a5" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 40801 } }
I20260812 05:20:40.632805 8312 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3991b6671a674e5fa40e4e19e4ae98a5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3991b6671a674e5fa40e4e19e4ae98a5. Latest consensus state: current_term: 1 leader_uuid: "3991b6671a674e5fa40e4e19e4ae98a5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3991b6671a674e5fa40e4e19e4ae98a5" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 40801 } } }
I20260812 05:20:40.632884 8312 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3991b6671a674e5fa40e4e19e4ae98a5 [sys.catalog]: This master's current role is: LEADER
I20260812 05:20:40.632817 8316 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3991b6671a674e5fa40e4e19e4ae98a5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3991b6671a674e5fa40e4e19e4ae98a5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3991b6671a674e5fa40e4e19e4ae98a5" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 40801 } } }
I20260812 05:20:40.633057 8316 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3991b6671a674e5fa40e4e19e4ae98a5 [sys.catalog]: This master's current role is: LEADER
I20260812 05:20:40.633710 8322 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 05:20:40.634232 8322 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 05:20:40.635782 8322 catalog_manager.cc:1347] Generated new cluster ID: 3e9f836a17dc487f8b330ba169377af1
I20260812 05:20:40.635836 8322 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 05:20:40.637693 8310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:40.631712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41228 (local address 127.31.100.62:40801)
0812 05:20:40.631947 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0812 05:20:40.631956 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 05:20:40.631991 (+ 35us) server_negotiation.cc:407] Connection header received
0812 05:20:40.632100 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:20:40.632107 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:20:40.632176 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:20:40.632321 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:20:40.632689 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:40.634326 (+ 1637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:40.635241 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:40.635782 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:40.635992 (+ 210us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:40.636843 (+ 851us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 05:20:40.636848 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 05:20:40.636848 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 05:20:40.637066 (+ 218us) server_negotiation.cc:922] calling sasl_server_start()
0812 05:20:40.637109 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 05:20:40.637124 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 05:20:40.637197 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0812 05:20:40.637254 (+ 57us) server_negotiation.cc:299] Negotiation successful
0812 05:20:40.637460 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":92,"threads_started":1}
I20260812 05:20:40.639744 32144 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 05:20:40.640362 8322 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 05:20:40.641026 8322 catalog_manager.cc:1504] Loading token signing keys...
I20260812 05:20:40.657704 8322 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3991b6671a674e5fa40e4e19e4ae98a5: Generated new TSK 0
I20260812 05:20:40.657887 8322 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260812 05:20:40.678568 8349 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260812 05:20:40.678697 8349 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
W20260812 05:20:40.679258 8347 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 05:20:40.663213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.62:40801 (local address 127.0.0.1:41234)
0812 05:20:40.663412 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0812 05:20:40.663421 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0812 05:20:40.663482 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:20:40.663649 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:20:40.663650 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:20:40.663656 (+ 6us) client_negotiation.cc:189] Negotiated authn=JWT
0812 05:20:40.663711 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:20:40.663712 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:20:40.664629 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:20:40.664630 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:20:40.665226 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:20:40.665227 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:20:40.665730 (+ 503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:20:40.665731 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:20:40.665791 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:40.665813 (+ 22us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0812 05:20:40.678829 (+ 13016us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41091/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0812 05:20:40.678942 (+ 113us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.31.100.62:40801: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41091/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":81,"threads_started":1}
I20260812 05:20:40.679694 8310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:40.663285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41234 (local address 127.31.100.62:40801)
0812 05:20:40.663329 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0812 05:20:40.663330 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0812 05:20:40.663487 (+ 157us) server_negotiation.cc:407] Connection header received
0812 05:20:40.663560 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:20:40.663561 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:20:40.663575 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:20:40.663648 (+ 73us) server_negotiation.cc:226] Negotiated authn=JWT
0812 05:20:40.663766 (+ 118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:40.664541 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:40.665303 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:40.665619 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:40.665692 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:40.665931 (+ 239us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0812 05:20:40.678679 (+ 12748us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41091/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0812 05:20:40.679505 (+ 826us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:41234: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41091/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":22}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.31.100.62:40801: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41091/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
W20260812 05:20:40.679754 8310 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:41234: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41091/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
I20260812 05:20:40.679772 32144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0waFvT/build/release/bin/kudu with pid 8201
2026-08-12T05:20:40Z chronyd exiting
I20260812 05:20:40.723212 32144 test_util.cc:175] -----------------------------------------------
I20260812 05:20:40.723285 32144 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786512017613366-32144-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (217 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260812 05:20:40.724017 32144 mini_oidc.cc:152] Starting JWKS server
I20260812 05:20:40.724655 32144 webserver.cc:460] Webserver started at https://127.0.0.1:39517/ using document root <none> and password file <none>
I20260812 05:20:40.724694 32144 mini_oidc.cc:189] Starting OIDC Discovery server
I20260812 05:20:40.724797 32144 webserver.cc:460] Webserver started at http://127.0.0.1:35455/ using document root <none> and password file <none>
2026-08-12T05:20:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T05:20:40Z Disabled control of system clock
I20260812 05:20:40.732873 32144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0waFvT/build/release/bin/kudu
/tmp/dist-test-task0waFvT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786512017613366-32144-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786512017613366-32144-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786512017613366-32144-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786512017613366-32144-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.100.62:41419
--webserver_interface=127.31.100.62
--webserver_port=0
--builtin_ntp_servers=127.31.100.20:45813
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.100.62:41419
--enable_jwt_token_auth=true
--jwks_url=https://localhost:39517/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786512017613366-32144-0/testchainca.cert with env {}
W20260812 05:20:40.811811 8357 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 05:20:40.812000 8357 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 05:20:40.812021 8357 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 05:20:40.813100 8357 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 05:20:40.813135 8357 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 05:20:40.813146 8357 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 05:20:40.813158 8357 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 05:20:40.813169 8357 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260812 05:20:40.813179 8357 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260812 05:20:40.813194 8357 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:39517/jwks.json
I20260812 05:20:40.814661 8357 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.100.20:45813
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786512017613366-32144-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786512017613366-32144-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.100.62:41419
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.100.62:41419
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:39517/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786512017613366-32144-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.100.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786512017613366-32144-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8357
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786512017613366-32144-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 04:20:34 UTC on ccycloud.quasar-phqdqp.root.comops.site
build id 4581871
I20260812 05:20:40.814851 8357 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 05:20:40.815014 8357 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 05:20:40.819577 8368 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 05:20:40.819586 8372 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 05:20:40.819614 8357 server_base.cc:1034] running on GCE node
W20260812 05:20:40.819598 8365 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 05:20:40.819991 8357 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 05:20:40.820245 8357 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 05:20:40.821420 8357 hybrid_clock.cc:648] HybridClock initialized: now 1786512040821404 us; error 27 us; skew 500 ppm
I20260812 05:20:40.822664 8357 webserver.cc:460] Webserver started at http://127.31.100.62:46277/ using document root <none> and password file <none>
I20260812 05:20:40.822889 8357 fs_manager.cc:359] Metadata directory not provided
I20260812 05:20:40.822938 8357 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 05:20:40.823061 8357 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 05:20:40.824057 8357 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786512017613366-32144-0/minicluster-data/master-0/data/instance:
uuid: "f8c5d6ea9da643e5b50abb5dfe9eb812"
format_stamp: "Formatted at 2026-08-12 05:20:40 on dist-test-slave-42z9"
server_key: "4b9d0772d6b15fb9b84b5d82e0e5ac25"
server_key_iv: "85d55a5c1f141041e448464bc51f7d2c"
server_key_version: "encryptionkey@0"
I20260812 05:20:40.824395 8357 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786512017613366-32144-0/minicluster-data/master-0/wal/instance:
uuid: "f8c5d6ea9da643e5b50abb5dfe9eb812"
format_stamp: "Formatted at 2026-08-12 05:20:40 on dist-test-slave-42z9"
server_key: "4b9d0772d6b15fb9b84b5d82e0e5ac25"
server_key_iv: "85d55a5c1f141041e448464bc51f7d2c"
server_key_version: "encryptionkey@0"
I20260812 05:20:40.825618 8357 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260812 05:20:40.826241 8380 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 05:20:40.826412 8357 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260812 05:20:40.826512 8357 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786512017613366-32144-0/minicluster-data/master-0/data,/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786512017613366-32144-0/minicluster-data/master-0/wal
uuid: "f8c5d6ea9da643e5b50abb5dfe9eb812"
format_stamp: "Formatted at 2026-08-12 05:20:40 on dist-test-slave-42z9"
server_key: "4b9d0772d6b15fb9b84b5d82e0e5ac25"
server_key_iv: "85d55a5c1f141041e448464bc51f7d2c"
server_key_version: "encryptionkey@0"
I20260812 05:20:40.826578 8357 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786512017613366-32144-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786512017613366-32144-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786512017613366-32144-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 05:20:40.842280 8357 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 05:20:40.842543 8357 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 05:20:40.842666 8357 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 05:20:40.846532 8357 rpc_server.cc:307] RPC server started. Bound to: 127.31.100.62:41419
I20260812 05:20:40.846566 8464 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.100.62:41419 every 8 connection(s)
I20260812 05:20:40.846860 8357 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786512017613366-32144-0/minicluster-data/master-0/data/info.pb
I20260812 05:20:40.847522 8467 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 05:20:40.850085 8467 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f8c5d6ea9da643e5b50abb5dfe9eb812: Bootstrap starting.
I20260812 05:20:40.850605 8467 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f8c5d6ea9da643e5b50abb5dfe9eb812: Neither blocks nor log segments found. Creating new log.
I20260812 05:20:40.850826 8467 log.cc:824] T 00000000000000000000000000000000 P f8c5d6ea9da643e5b50abb5dfe9eb812: Log is configured to *not* fsync() on all Append() calls
I20260812 05:20:40.851433 8467 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f8c5d6ea9da643e5b50abb5dfe9eb812: No bootstrap required, opened a new log
I20260812 05:20:40.852959 8467 raft_consensus.cc:348] T 00000000000000000000000000000000 P f8c5d6ea9da643e5b50abb5dfe9eb812 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8c5d6ea9da643e5b50abb5dfe9eb812" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 41419 } }
I20260812 05:20:40.853097 8467 raft_consensus.cc:374] T 00000000000000000000000000000000 P f8c5d6ea9da643e5b50abb5dfe9eb812 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 05:20:40.853118 8467 raft_consensus.cc:729] T 00000000000000000000000000000000 P f8c5d6ea9da643e5b50abb5dfe9eb812 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8c5d6ea9da643e5b50abb5dfe9eb812, State: Initialized, Role: FOLLOWER
I20260812 05:20:40.853214 8467 consensus_queue.cc:260] T 00000000000000000000000000000000 P f8c5d6ea9da643e5b50abb5dfe9eb812 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8c5d6ea9da643e5b50abb5dfe9eb812" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 41419 } }
I20260812 05:20:40.853261 8467 raft_consensus.cc:388] T 00000000000000000000000000000000 P f8c5d6ea9da643e5b50abb5dfe9eb812 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 05:20:40.853284 8467 raft_consensus.cc:482] T 00000000000000000000000000000000 P f8c5d6ea9da643e5b50abb5dfe9eb812 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 05:20:40.853313 8467 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f8c5d6ea9da643e5b50abb5dfe9eb812 [term 0 FOLLOWER]: Advancing to term 1
I20260812 05:20:40.853819 8467 raft_consensus.cc:504] T 00000000000000000000000000000000 P f8c5d6ea9da643e5b50abb5dfe9eb812 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8c5d6ea9da643e5b50abb5dfe9eb812" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 41419 } }
I20260812 05:20:40.853907 8467 leader_election.cc:302] T 00000000000000000000000000000000 P f8c5d6ea9da643e5b50abb5dfe9eb812 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f8c5d6ea9da643e5b50abb5dfe9eb812; no voters:
I20260812 05:20:40.854029 8467 leader_election.cc:288] T 00000000000000000000000000000000 P f8c5d6ea9da643e5b50abb5dfe9eb812 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 05:20:40.854074 8472 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f8c5d6ea9da643e5b50abb5dfe9eb812 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 05:20:40.854240 8472 raft_consensus.cc:686] T 00000000000000000000000000000000 P f8c5d6ea9da643e5b50abb5dfe9eb812 [term 1 LEADER]: Becoming Leader. State: Replica: f8c5d6ea9da643e5b50abb5dfe9eb812, State: Running, Role: LEADER
I20260812 05:20:40.854247 8467 sys_catalog.cc:564] T 00000000000000000000000000000000 P f8c5d6ea9da643e5b50abb5dfe9eb812 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 05:20:40.854342 8472 consensus_queue.cc:237] T 00000000000000000000000000000000 P f8c5d6ea9da643e5b50abb5dfe9eb812 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8c5d6ea9da643e5b50abb5dfe9eb812" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 41419 } }
I20260812 05:20:40.854699 8473 sys_catalog.cc:455] T 00000000000000000000000000000000 P f8c5d6ea9da643e5b50abb5dfe9eb812 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f8c5d6ea9da643e5b50abb5dfe9eb812" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8c5d6ea9da643e5b50abb5dfe9eb812" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 41419 } } }
I20260812 05:20:40.854817 8473 sys_catalog.cc:458] T 00000000000000000000000000000000 P f8c5d6ea9da643e5b50abb5dfe9eb812 [sys.catalog]: This master's current role is: LEADER
I20260812 05:20:40.854828 8474 sys_catalog.cc:455] T 00000000000000000000000000000000 P f8c5d6ea9da643e5b50abb5dfe9eb812 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f8c5d6ea9da643e5b50abb5dfe9eb812. Latest consensus state: current_term: 1 leader_uuid: "f8c5d6ea9da643e5b50abb5dfe9eb812" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8c5d6ea9da643e5b50abb5dfe9eb812" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 41419 } } }
I20260812 05:20:40.854918 8474 sys_catalog.cc:458] T 00000000000000000000000000000000 P f8c5d6ea9da643e5b50abb5dfe9eb812 [sys.catalog]: This master's current role is: LEADER
I20260812 05:20:40.855106 8483 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 05:20:40.855592 8483 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 05:20:40.856941 32144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0waFvT/build/release/bin/kudu as pid 8357
I20260812 05:20:40.857019 8483 catalog_manager.cc:1347] Generated new cluster ID: fd6cf7f449bf4e65bcc111d7f5b6b9ad
I20260812 05:20:40.857054 32144 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786512017613366-32144-0/minicluster-data/master-0/wal/instance
I20260812 05:20:40.857064 8483 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 05:20:40.857232 32144 external_mini_cluster.cc:1428] Setting key 61b72d58fc9b7593926177a8cacf860f
I20260812 05:20:40.862869 8502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:40.857653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57718 (local address 127.31.100.62:41419)
0812 05:20:40.857832 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0812 05:20:40.857841 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 05:20:40.857871 (+ 30us) server_negotiation.cc:407] Connection header received
0812 05:20:40.857989 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:20:40.857996 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:20:40.858072 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:20:40.858220 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:20:40.858365 (+ 145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:40.859769 (+ 1404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:40.860798 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:40.861426 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:40.861573 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:40.862175 (+ 602us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 05:20:40.862180 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 05:20:40.862180 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 05:20:40.862393 (+ 213us) server_negotiation.cc:922] calling sasl_server_start()
0812 05:20:40.862437 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 05:20:40.862451 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 05:20:40.862518 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0812 05:20:40.862573 (+ 55us) server_negotiation.cc:299] Negotiation successful
0812 05:20:40.862655 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":70,"threads_started":1}
I20260812 05:20:40.863757 32144 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 05:20:40.874840 8483 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 05:20:40.875314 8483 catalog_manager.cc:1504] Loading token signing keys...
I20260812 05:20:40.879256 8483 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f8c5d6ea9da643e5b50abb5dfe9eb812: Generated new TSK 0
I20260812 05:20:40.879421 8483 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260812 05:20:40.908859 8514 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260812 05:20:40.908949 8514 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260812 05:20:40.909162 8502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:40.887997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57724 (local address 127.31.100.62:41419)
0812 05:20:40.888041 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0812 05:20:40.888043 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 05:20:40.888143 (+ 100us) server_negotiation.cc:407] Connection header received
0812 05:20:40.888206 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:20:40.888207 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:20:40.888218 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:20:40.888275 (+ 57us) server_negotiation.cc:226] Negotiated authn=JWT
0812 05:20:40.888390 (+ 115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:40.888878 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:40.889671 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:40.889868 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:40.889917 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:40.890076 (+ 159us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0812 05:20:40.908975 (+ 18899us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39517/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0812 05:20:40.909112 (+ 137us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:57724: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39517/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":24}
W20260812 05:20:40.909212 8502 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:57724: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39517/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260812 05:20:40.909276 8513 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 05:20:40.887951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.62:41419 (local address 127.0.0.1:57724)
0812 05:20:40.888085 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0812 05:20:40.888094 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0812 05:20:40.888151 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:20:40.888279 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:20:40.888280 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:20:40.888281 (+ 1us) client_negotiation.cc:189] Negotiated authn=JWT
0812 05:20:40.888335 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:20:40.888337 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:20:40.888954 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:20:40.888955 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:20:40.889594 (+ 639us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:20:40.889597 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:20:40.889923 (+ 326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:20:40.889924 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:20:40.889991 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:40.889996 (+ 5us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0812 05:20:40.909079 (+ 19083us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39517/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0812 05:20:40.909184 (+ 105us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.31.100.62:41419: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39517/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":86,"threads_started":1}
I20260812 05:20:40.909637 32144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0waFvT/build/release/bin/kudu with pid 8357
2026-08-12T05:20:40Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (237 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260812 05:20:40.961261 32144 mini_oidc.cc:152] Starting JWKS server
I20260812 05:20:40.962011 32144 webserver.cc:460] Webserver started at https://127.0.0.1:46419/ using document root <none> and password file <none>
I20260812 05:20:40.962059 32144 mini_oidc.cc:189] Starting OIDC Discovery server
I20260812 05:20:40.962198 32144 webserver.cc:460] Webserver started at http://127.0.0.1:39055/ using document root <none> and password file <none>
2026-08-12T05:20:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T05:20:40Z Disabled control of system clock
I20260812 05:20:40.970449 32144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0waFvT/build/release/bin/kudu
/tmp/dist-test-task0waFvT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786512017613366-32144-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786512017613366-32144-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786512017613366-32144-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786512017613366-32144-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.100.62:40853
--webserver_interface=127.31.100.62
--webserver_port=0
--builtin_ntp_servers=127.31.100.20:34371
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.100.62:40853
--enable_jwt_token_auth=true
--jwks_url=https://localhost:46419/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260812 05:20:41.049587 8524 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 05:20:41.049784 8524 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 05:20:41.049805 8524 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 05:20:41.050905 8524 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 05:20:41.050940 8524 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 05:20:41.050951 8524 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 05:20:41.050961 8524 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 05:20:41.050972 8524 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260812 05:20:41.050983 8524 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260812 05:20:41.050993 8524 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:46419/jwks.json
I20260812 05:20:41.052266 8524 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.100.20:34371
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786512017613366-32144-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786512017613366-32144-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.100.62:40853
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.100.62:40853
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:46419/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786512017613366-32144-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.100.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8524
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786512017613366-32144-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 04:20:34 UTC on ccycloud.quasar-phqdqp.root.comops.site
build id 4581871
I20260812 05:20:41.052461 8524 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 05:20:41.052697 8524 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 05:20:41.057178 8536 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 05:20:41.057262 8539 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 05:20:41.057372 8524 server_base.cc:1034] running on GCE node
W20260812 05:20:41.057269 8537 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 05:20:41.057720 8524 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 05:20:41.057992 8524 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 05:20:41.059142 8524 hybrid_clock.cc:648] HybridClock initialized: now 1786512041059128 us; error 33 us; skew 500 ppm
I20260812 05:20:41.060335 8524 webserver.cc:460] Webserver started at http://127.31.100.62:39887/ using document root <none> and password file <none>
I20260812 05:20:41.060595 8524 fs_manager.cc:359] Metadata directory not provided
I20260812 05:20:41.060647 8524 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 05:20:41.060776 8524 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 05:20:41.061911 8524 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786512017613366-32144-0/minicluster-data/master-0/data/instance:
uuid: "cc7462b174594a95b13ef006db4162b6"
format_stamp: "Formatted at 2026-08-12 05:20:41 on dist-test-slave-42z9"
server_key: "f240f36ee12e10079ef5ee431bcae90c"
server_key_iv: "e30180b49219429a723f85a98bacf799"
server_key_version: "encryptionkey@0"
I20260812 05:20:41.062282 8524 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786512017613366-32144-0/minicluster-data/master-0/wal/instance:
uuid: "cc7462b174594a95b13ef006db4162b6"
format_stamp: "Formatted at 2026-08-12 05:20:41 on dist-test-slave-42z9"
server_key: "f240f36ee12e10079ef5ee431bcae90c"
server_key_iv: "e30180b49219429a723f85a98bacf799"
server_key_version: "encryptionkey@0"
I20260812 05:20:41.063593 8524 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260812 05:20:41.064280 8550 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 05:20:41.064433 8524 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260812 05:20:41.064534 8524 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786512017613366-32144-0/minicluster-data/master-0/data,/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786512017613366-32144-0/minicluster-data/master-0/wal
uuid: "cc7462b174594a95b13ef006db4162b6"
format_stamp: "Formatted at 2026-08-12 05:20:41 on dist-test-slave-42z9"
server_key: "f240f36ee12e10079ef5ee431bcae90c"
server_key_iv: "e30180b49219429a723f85a98bacf799"
server_key_version: "encryptionkey@0"
I20260812 05:20:41.064596 8524 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786512017613366-32144-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786512017613366-32144-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786512017613366-32144-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 05:20:41.081899 8524 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 05:20:41.082168 8524 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 05:20:41.082288 8524 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 05:20:41.086097 8524 rpc_server.cc:307] RPC server started. Bound to: 127.31.100.62:40853
I20260812 05:20:41.086133 8618 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.100.62:40853 every 8 connection(s)
I20260812 05:20:41.086444 8524 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786512017613366-32144-0/minicluster-data/master-0/data/info.pb
I20260812 05:20:41.087100 8619 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 05:20:41.089692 8619 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cc7462b174594a95b13ef006db4162b6: Bootstrap starting.
I20260812 05:20:41.090435 8619 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cc7462b174594a95b13ef006db4162b6: Neither blocks nor log segments found. Creating new log.
I20260812 05:20:41.090705 8619 log.cc:824] T 00000000000000000000000000000000 P cc7462b174594a95b13ef006db4162b6: Log is configured to *not* fsync() on all Append() calls
I20260812 05:20:41.091404 8619 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cc7462b174594a95b13ef006db4162b6: No bootstrap required, opened a new log
I20260812 05:20:41.093001 8619 raft_consensus.cc:348] T 00000000000000000000000000000000 P cc7462b174594a95b13ef006db4162b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc7462b174594a95b13ef006db4162b6" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 40853 } }
I20260812 05:20:41.093137 8619 raft_consensus.cc:374] T 00000000000000000000000000000000 P cc7462b174594a95b13ef006db4162b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 05:20:41.093158 8619 raft_consensus.cc:729] T 00000000000000000000000000000000 P cc7462b174594a95b13ef006db4162b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc7462b174594a95b13ef006db4162b6, State: Initialized, Role: FOLLOWER
I20260812 05:20:41.093290 8619 consensus_queue.cc:260] T 00000000000000000000000000000000 P cc7462b174594a95b13ef006db4162b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc7462b174594a95b13ef006db4162b6" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 40853 } }
I20260812 05:20:41.093350 8619 raft_consensus.cc:388] T 00000000000000000000000000000000 P cc7462b174594a95b13ef006db4162b6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 05:20:41.093374 8619 raft_consensus.cc:482] T 00000000000000000000000000000000 P cc7462b174594a95b13ef006db4162b6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 05:20:41.093416 8619 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cc7462b174594a95b13ef006db4162b6 [term 0 FOLLOWER]: Advancing to term 1
I20260812 05:20:41.093961 8619 raft_consensus.cc:504] T 00000000000000000000000000000000 P cc7462b174594a95b13ef006db4162b6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc7462b174594a95b13ef006db4162b6" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 40853 } }
I20260812 05:20:41.094048 8619 leader_election.cc:302] T 00000000000000000000000000000000 P cc7462b174594a95b13ef006db4162b6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cc7462b174594a95b13ef006db4162b6; no voters:
I20260812 05:20:41.094192 8619 leader_election.cc:288] T 00000000000000000000000000000000 P cc7462b174594a95b13ef006db4162b6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 05:20:41.094249 8625 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cc7462b174594a95b13ef006db4162b6 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 05:20:41.094419 8619 sys_catalog.cc:564] T 00000000000000000000000000000000 P cc7462b174594a95b13ef006db4162b6 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 05:20:41.094451 8625 raft_consensus.cc:686] T 00000000000000000000000000000000 P cc7462b174594a95b13ef006db4162b6 [term 1 LEADER]: Becoming Leader. State: Replica: cc7462b174594a95b13ef006db4162b6, State: Running, Role: LEADER
I20260812 05:20:41.094484 32144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0waFvT/build/release/bin/kudu as pid 8524
I20260812 05:20:41.094579 32144 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786512017613366-32144-0/minicluster-data/master-0/wal/instance
I20260812 05:20:41.094660 8625 consensus_queue.cc:237] T 00000000000000000000000000000000 P cc7462b174594a95b13ef006db4162b6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc7462b174594a95b13ef006db4162b6" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 40853 } }
I20260812 05:20:41.094790 32144 external_mini_cluster.cc:1428] Setting key d86ad944cb043a2db4dfc46931e0c326
I20260812 05:20:41.095017 8627 sys_catalog.cc:455] T 00000000000000000000000000000000 P cc7462b174594a95b13ef006db4162b6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cc7462b174594a95b13ef006db4162b6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc7462b174594a95b13ef006db4162b6" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 40853 } } }
I20260812 05:20:41.095093 8627 sys_catalog.cc:458] T 00000000000000000000000000000000 P cc7462b174594a95b13ef006db4162b6 [sys.catalog]: This master's current role is: LEADER
I20260812 05:20:41.095525 8629 sys_catalog.cc:455] T 00000000000000000000000000000000 P cc7462b174594a95b13ef006db4162b6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cc7462b174594a95b13ef006db4162b6. Latest consensus state: current_term: 1 leader_uuid: "cc7462b174594a95b13ef006db4162b6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc7462b174594a95b13ef006db4162b6" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 40853 } } }
I20260812 05:20:41.095566 8640 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 05:20:41.095641 8629 sys_catalog.cc:458] T 00000000000000000000000000000000 P cc7462b174594a95b13ef006db4162b6 [sys.catalog]: This master's current role is: LEADER
I20260812 05:20:41.096120 8640 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 05:20:41.097599 8640 catalog_manager.cc:1347] Generated new cluster ID: c88c1e12f0fc4e649dd4d020edfe4ab4
I20260812 05:20:41.097653 8640 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 05:20:41.101244 8642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:41.095217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59326 (local address 127.31.100.62:40853)
0812 05:20:41.095891 (+ 674us) server_negotiation.cc:206] Beginning negotiation
0812 05:20:41.095900 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 05:20:41.095936 (+ 36us) server_negotiation.cc:407] Connection header received
0812 05:20:41.096052 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:20:41.096060 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:20:41.096128 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:20:41.096541 (+ 413us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:20:41.096643 (+ 102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:41.098013 (+ 1370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:41.099060 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:41.099683 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:41.099808 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:41.100429 (+ 621us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 05:20:41.100433 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 05:20:41.100433 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 05:20:41.100662 (+ 229us) server_negotiation.cc:922] calling sasl_server_start()
0812 05:20:41.100708 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 05:20:41.100723 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 05:20:41.100794 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0812 05:20:41.100848 (+ 54us) server_negotiation.cc:299] Negotiation successful
0812 05:20:41.101031 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":583,"thread_start_us":71,"threads_started":1}
I20260812 05:20:41.102109 32144 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 05:20:41.134042 8640 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 05:20:41.134557 8640 catalog_manager.cc:1504] Loading token signing keys...
I20260812 05:20:41.139954 8640 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cc7462b174594a95b13ef006db4162b6: Generated new TSK 0
I20260812 05:20:41.140118 8640 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 05:20:41.159886 8642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:41.155541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59336 (local address 127.31.100.62:40853)
0812 05:20:41.155867 (+ 326us) server_negotiation.cc:206] Beginning negotiation
0812 05:20:41.155869 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 05:20:41.156003 (+ 134us) server_negotiation.cc:407] Connection header received
0812 05:20:41.156117 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:20:41.156119 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:20:41.156132 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:20:41.156205 (+ 73us) server_negotiation.cc:226] Negotiated authn=JWT
0812 05:20:41.156347 (+ 142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:41.157072 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:41.158229 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:41.158528 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:41.158597 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:41.158814 (+ 217us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0812 05:20:41.159693 (+ 879us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46419/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0812 05:20:41.159821 (+ 128us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:59336: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46419/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":300}
W20260812 05:20:41.159953 8642 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:59336: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46419/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260812 05:20:41.159973 8670 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 05:20:41.155432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.62:40853 (local address 127.0.0.1:59336)
0812 05:20:41.155930 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0812 05:20:41.155941 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0812 05:20:41.156031 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:20:41.156211 (+ 180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:20:41.156212 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:20:41.156215 (+ 3us) client_negotiation.cc:189] Negotiated authn=JWT
0812 05:20:41.156286 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:20:41.156288 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:20:41.157172 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:20:41.157173 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:20:41.158123 (+ 950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:20:41.158126 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:20:41.158635 (+ 509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:20:41.158637 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:20:41.158710 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:41.158716 (+ 6us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0812 05:20:41.159780 (+ 1064us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46419/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0812 05:20:41.159871 (+ 91us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.31.100.62:40853: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46419/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":471,"thread_start_us":81,"threads_started":1}
I20260812 05:20:41.160228 32144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0waFvT/build/release/bin/kudu with pid 8524
E20260812 05:20:41.175756 8672 webserver.cc:533] Webserver: sslize failed:
E20260812 05:20:41.175829 8672 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-08-12T05:20:41Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (244 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-08-12T05:20:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T05:20:41Z Disabled control of system clock
I20260812 05:20:41.213548 32144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0waFvT/build/release/bin/kudu
/tmp/dist-test-task0waFvT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786512017613366-32144-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786512017613366-32144-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786512017613366-32144-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786512017613366-32144-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.100.62:43291
--webserver_interface=127.31.100.62
--webserver_port=0
--builtin_ntp_servers=127.31.100.20:40685
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.100.62:43291
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786512017613366-32144-0/testchainca.cert with env {}
W20260812 05:20:41.293295 8678 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 05:20:41.293506 8678 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 05:20:41.293529 8678 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 05:20:41.294613 8678 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 05:20:41.294646 8678 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 05:20:41.294658 8678 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 05:20:41.294669 8678 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 05:20:41.294680 8678 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260812 05:20:41.294696 8678 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260812 05:20:41.294708 8678 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260812 05:20:41.296129 8678 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.100.20:40685
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786512017613366-32144-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786512017613366-32144-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.100.62:43291
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.100.62:43291
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786512017613366-32144-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.100.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786512017613366-32144-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8678
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786512017613366-32144-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 04:20:34 UTC on ccycloud.quasar-phqdqp.root.comops.site
build id 4581871
I20260812 05:20:41.296314 8678 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 05:20:41.296473 8678 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 05:20:41.301012 8686 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 05:20:41.301039 8684 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 05:20:41.301208 8678 server_base.cc:1034] running on GCE node
W20260812 05:20:41.301039 8689 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 05:20:41.301527 8678 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 05:20:41.301802 8678 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 05:20:41.302953 8678 hybrid_clock.cc:648] HybridClock initialized: now 1786512041302928 us; error 30 us; skew 500 ppm
I20260812 05:20:41.304057 8678 webserver.cc:460] Webserver started at http://127.31.100.62:45803/ using document root <none> and password file <none>
I20260812 05:20:41.304260 8678 fs_manager.cc:359] Metadata directory not provided
I20260812 05:20:41.304307 8678 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 05:20:41.304426 8678 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 05:20:41.305539 8678 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786512017613366-32144-0/minicluster-data/master-0/data/instance:
uuid: "32af9fd634274617a39c839e3c788b93"
format_stamp: "Formatted at 2026-08-12 05:20:41 on dist-test-slave-42z9"
server_key: "e6f55eea66ad85657965b88090e5d5ad"
server_key_iv: "88b7a1710220450cfd0f641deaaba4dd"
server_key_version: "encryptionkey@0"
I20260812 05:20:41.305900 8678 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786512017613366-32144-0/minicluster-data/master-0/wal/instance:
uuid: "32af9fd634274617a39c839e3c788b93"
format_stamp: "Formatted at 2026-08-12 05:20:41 on dist-test-slave-42z9"
server_key: "e6f55eea66ad85657965b88090e5d5ad"
server_key_iv: "88b7a1710220450cfd0f641deaaba4dd"
server_key_version: "encryptionkey@0"
I20260812 05:20:41.307140 8678 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260812 05:20:41.307767 8699 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 05:20:41.307909 8678 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260812 05:20:41.308000 8678 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786512017613366-32144-0/minicluster-data/master-0/data,/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786512017613366-32144-0/minicluster-data/master-0/wal
uuid: "32af9fd634274617a39c839e3c788b93"
format_stamp: "Formatted at 2026-08-12 05:20:41 on dist-test-slave-42z9"
server_key: "e6f55eea66ad85657965b88090e5d5ad"
server_key_iv: "88b7a1710220450cfd0f641deaaba4dd"
server_key_version: "encryptionkey@0"
I20260812 05:20:41.308058 8678 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786512017613366-32144-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786512017613366-32144-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786512017613366-32144-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 05:20:41.320612 8678 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 05:20:41.320847 8678 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 05:20:41.320967 8678 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 05:20:41.324908 8678 rpc_server.cc:307] RPC server started. Bound to: 127.31.100.62:43291
I20260812 05:20:41.324949 8772 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.100.62:43291 every 8 connection(s)
I20260812 05:20:41.325227 8678 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786512017613366-32144-0/minicluster-data/master-0/data/info.pb
I20260812 05:20:41.326161 8773 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 05:20:41.327437 32144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0waFvT/build/release/bin/kudu as pid 8678
I20260812 05:20:41.327527 32144 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786512017613366-32144-0/minicluster-data/master-0/wal/instance
I20260812 05:20:41.327679 32144 external_mini_cluster.cc:1428] Setting key ccdf74c04c87af4f534f92aabacfff87
I20260812 05:20:41.328593 8773 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 32af9fd634274617a39c839e3c788b93: Bootstrap starting.
I20260812 05:20:41.329267 8773 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 32af9fd634274617a39c839e3c788b93: Neither blocks nor log segments found. Creating new log.
I20260812 05:20:41.329591 8773 log.cc:824] T 00000000000000000000000000000000 P 32af9fd634274617a39c839e3c788b93: Log is configured to *not* fsync() on all Append() calls
I20260812 05:20:41.330221 8773 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 32af9fd634274617a39c839e3c788b93: No bootstrap required, opened a new log
I20260812 05:20:41.332439 8773 raft_consensus.cc:348] T 00000000000000000000000000000000 P 32af9fd634274617a39c839e3c788b93 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32af9fd634274617a39c839e3c788b93" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 43291 } }
I20260812 05:20:41.332567 8773 raft_consensus.cc:374] T 00000000000000000000000000000000 P 32af9fd634274617a39c839e3c788b93 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 05:20:41.332592 8773 raft_consensus.cc:729] T 00000000000000000000000000000000 P 32af9fd634274617a39c839e3c788b93 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32af9fd634274617a39c839e3c788b93, State: Initialized, Role: FOLLOWER
I20260812 05:20:41.332715 8773 consensus_queue.cc:260] T 00000000000000000000000000000000 P 32af9fd634274617a39c839e3c788b93 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32af9fd634274617a39c839e3c788b93" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 43291 } }
I20260812 05:20:41.332813 8773 raft_consensus.cc:388] T 00000000000000000000000000000000 P 32af9fd634274617a39c839e3c788b93 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 05:20:41.332855 8773 raft_consensus.cc:482] T 00000000000000000000000000000000 P 32af9fd634274617a39c839e3c788b93 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 05:20:41.332914 8773 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 32af9fd634274617a39c839e3c788b93 [term 0 FOLLOWER]: Advancing to term 1
I20260812 05:20:41.333062 8777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:41.328121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55090 (local address 127.31.100.62:43291)
0812 05:20:41.328295 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0812 05:20:41.328302 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 05:20:41.328325 (+ 23us) server_negotiation.cc:407] Connection header received
0812 05:20:41.328737 (+ 412us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:20:41.328744 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:20:41.328803 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:20:41.328917 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:20:41.329184 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:41.330190 (+ 1006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:41.331227 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:41.331648 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:41.331730 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:41.332337 (+ 607us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 05:20:41.332340 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 05:20:41.332340 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 05:20:41.332508 (+ 168us) server_negotiation.cc:922] calling sasl_server_start()
0812 05:20:41.332540 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 05:20:41.332550 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 05:20:41.332606 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0812 05:20:41.332704 (+ 98us) server_negotiation.cc:299] Negotiation successful
0812 05:20:41.332784 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108,"thread_start_us":70,"threads_started":1}
I20260812 05:20:41.333532 8773 raft_consensus.cc:504] T 00000000000000000000000000000000 P 32af9fd634274617a39c839e3c788b93 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32af9fd634274617a39c839e3c788b93" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 43291 } }
I20260812 05:20:41.333640 8773 leader_election.cc:302] T 00000000000000000000000000000000 P 32af9fd634274617a39c839e3c788b93 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 32af9fd634274617a39c839e3c788b93; no voters:
I20260812 05:20:41.333796 8773 leader_election.cc:288] T 00000000000000000000000000000000 P 32af9fd634274617a39c839e3c788b93 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 05:20:41.333971 8780 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 32af9fd634274617a39c839e3c788b93 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 05:20:41.334024 8773 sys_catalog.cc:564] T 00000000000000000000000000000000 P 32af9fd634274617a39c839e3c788b93 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 05:20:41.334110 8780 raft_consensus.cc:686] T 00000000000000000000000000000000 P 32af9fd634274617a39c839e3c788b93 [term 1 LEADER]: Becoming Leader. State: Replica: 32af9fd634274617a39c839e3c788b93, State: Running, Role: LEADER
I20260812 05:20:41.334187 32144 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 05:20:41.334192 8780 consensus_queue.cc:237] T 00000000000000000000000000000000 P 32af9fd634274617a39c839e3c788b93 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32af9fd634274617a39c839e3c788b93" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 43291 } }
I20260812 05:20:41.334486 8780 sys_catalog.cc:455] T 00000000000000000000000000000000 P 32af9fd634274617a39c839e3c788b93 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 32af9fd634274617a39c839e3c788b93. Latest consensus state: current_term: 1 leader_uuid: "32af9fd634274617a39c839e3c788b93" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32af9fd634274617a39c839e3c788b93" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 43291 } } }
I20260812 05:20:41.334496 8782 sys_catalog.cc:455] T 00000000000000000000000000000000 P 32af9fd634274617a39c839e3c788b93 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "32af9fd634274617a39c839e3c788b93" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32af9fd634274617a39c839e3c788b93" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 43291 } } }
I20260812 05:20:41.334558 8780 sys_catalog.cc:458] T 00000000000000000000000000000000 P 32af9fd634274617a39c839e3c788b93 [sys.catalog]: This master's current role is: LEADER
I20260812 05:20:41.334559 8782 sys_catalog.cc:458] T 00000000000000000000000000000000 P 32af9fd634274617a39c839e3c788b93 [sys.catalog]: This master's current role is: LEADER
W20260812 05:20:41.335947 8807 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 32af9fd634274617a39c839e3c788b93: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 05:20:41.335989 8807 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 05:20:41.336027 8791 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 05:20:41.336150 8791 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 05:20:41.337386 8791 catalog_manager.cc:1347] Generated new cluster ID: 19d9af120bde447086aca3f72131293e
I20260812 05:20:41.337450 8791 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 05:20:41.362622 8791 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 05:20:41.363108 8791 catalog_manager.cc:1504] Loading token signing keys...
I20260812 05:20:41.366184 8791 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 32af9fd634274617a39c839e3c788b93: Generated new TSK 0
I20260812 05:20:41.366336 8791 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 05:20:41.418097 8777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:41.375904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55092 (local address 127.31.100.62:43291)
0812 05:20:41.375956 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0812 05:20:41.375958 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 05:20:41.376056 (+ 98us) server_negotiation.cc:407] Connection header received
0812 05:20:41.376129 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:20:41.376129 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:20:41.376139 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:20:41.376200 (+ 61us) server_negotiation.cc:226] Negotiated authn=JWT
0812 05:20:41.376329 (+ 129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:41.376799 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:41.377956 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:41.378158 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:41.378210 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:41.378366 (+ 156us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0812 05:20:41.417897 (+ 39531us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0812 05:20:41.418041 (+ 144us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:55092: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":30}
W20260812 05:20:41.418161 8777 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:55092: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260812 05:20:41.418396 8819 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 05:20:41.375846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.62:43291 (local address 127.0.0.1:55092)
0812 05:20:41.375992 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0812 05:20:41.376002 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0812 05:20:41.376068 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:20:41.376207 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:20:41.376208 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:20:41.376210 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0812 05:20:41.376271 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:20:41.376273 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:20:41.376878 (+ 605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:20:41.376879 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:20:41.377882 (+ 1003us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:20:41.377885 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:20:41.378230 (+ 345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:20:41.378231 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:20:41.378287 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:41.378293 (+ 6us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0812 05:20:41.418047 (+ 39754us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0812 05:20:41.418278 (+ 231us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.31.100.62:43291: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":76,"threads_started":1}
I20260812 05:20:41.418748 32144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0waFvT/build/release/bin/kudu with pid 8678
2026-08-12T05:20:41Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (234 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (89 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (89 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786512017613366-32144-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 05:20:41 dist-test-slave-42z9 krb5kdc[8841](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 05:20:41 dist-test-slave-42z9 krb5kdc[8841](info): set up 2 sockets
Aug 12 05:20:41 dist-test-slave-42z9 krb5kdc[8841](info): commencing operation
krb5kdc: starting...
W20260812 05:20:42.758365 32144 mini_kdc.cc:121] Time spent starting KDC: real 1.139s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786512017613366-32144-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786512017613366-32144-0/krb5kdc/test-user.keytab.
Aug 12 05:20:42 dist-test-slave-42z9 krb5kdc[8841](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512042, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T05:20:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T05:20:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.100.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.100.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.100.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.62.keytab.
Entry for principal kudu/127.31.100.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.62.keytab.
I20260812 05:20:42.869845 32144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0waFvT/build/release/bin/kudu
/tmp/dist-test-task0waFvT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786512017613366-32144-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786512017613366-32144-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786512017613366-32144-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786512017613366-32144-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.100.62:34661
--webserver_interface=127.31.100.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.62.keytab
--principal=kudu/127.31.100.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.100.20:40397
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.100.62:34661
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786512017613366-32144-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786512017613366-32144-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786512017613366-32144-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 05:20:42.949174 8862 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 05:20:42.949365 8862 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 05:20:42.949385 8862 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 05:20:42.950549 8862 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 05:20:42.950589 8862 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 05:20:42.950604 8862 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 05:20:42.950615 8862 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 05:20:42.950628 8862 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 05:20:42.952080 8862 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.100.20:40397
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786512017613366-32144-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786512017613366-32144-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.100.62:34661
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.62.keytab
--principal=kudu/127.31.100.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.100.62:34661
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786512017613366-32144-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.100.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8862
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786512017613366-32144-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 04:20:34 UTC on ccycloud.quasar-phqdqp.root.comops.site
build id 4581871
I20260812 05:20:42.952299 8862 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 05:20:42.952497 8862 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 05:20:42.957094 8868 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 05:20:42.957136 8875 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 05:20:42.957171 8862 server_base.cc:1034] running on GCE node
W20260812 05:20:42.957159 8867 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 05:20:42.957564 8862 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 05:20:42.957830 8862 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 05:20:42.958978 8862 hybrid_clock.cc:648] HybridClock initialized: now 1786512042958967 us; error 29 us; skew 500 ppm
Aug 12 05:20:42 dist-test-slave-42z9 krb5kdc[8841](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512042, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.100.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 05:20:42.961338 8862 init.cc:377] Logged in from keytab as kudu/127.31.100.62@KRBTEST.COM (short username kudu)
I20260812 05:20:42.961807 8862 webserver.cc:460] Webserver started at http://127.31.100.62:46361/ using document root <none> and password file <none>
I20260812 05:20:42.962011 8862 fs_manager.cc:359] Metadata directory not provided
I20260812 05:20:42.962060 8862 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 05:20:42.962181 8862 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 05:20:42.963171 8862 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786512017613366-32144-0/minicluster-data/master-0/data/instance:
uuid: "ee14c40d691b4a77b1c4067fdfaa5732"
format_stamp: "Formatted at 2026-08-12 05:20:42 on dist-test-slave-42z9"
server_key: "7155e4f927bfb151ecca03ea315f0247"
server_key_iv: "13588dc78e9cb949c8b00367ad427e75"
server_key_version: "encryptionkey@0"
I20260812 05:20:42.963487 8862 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786512017613366-32144-0/minicluster-data/master-0/wal/instance:
uuid: "ee14c40d691b4a77b1c4067fdfaa5732"
format_stamp: "Formatted at 2026-08-12 05:20:42 on dist-test-slave-42z9"
server_key: "7155e4f927bfb151ecca03ea315f0247"
server_key_iv: "13588dc78e9cb949c8b00367ad427e75"
server_key_version: "encryptionkey@0"
I20260812 05:20:42.964713 8862 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260812 05:20:42.965361 8885 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 05:20:42.965520 8862 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260812 05:20:42.965607 8862 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786512017613366-32144-0/minicluster-data/master-0/data,/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786512017613366-32144-0/minicluster-data/master-0/wal
uuid: "ee14c40d691b4a77b1c4067fdfaa5732"
format_stamp: "Formatted at 2026-08-12 05:20:42 on dist-test-slave-42z9"
server_key: "7155e4f927bfb151ecca03ea315f0247"
server_key_iv: "13588dc78e9cb949c8b00367ad427e75"
server_key_version: "encryptionkey@0"
I20260812 05:20:42.965667 8862 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786512017613366-32144-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786512017613366-32144-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786512017613366-32144-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 05:20:42.992219 8862 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 05:20:42.994081 8862 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 05:20:42.994213 8862 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 05:20:42.998132 8862 rpc_server.cc:307] RPC server started. Bound to: 127.31.100.62:34661
I20260812 05:20:42.998173 8957 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.100.62:34661 every 8 connection(s)
I20260812 05:20:42.998485 8862 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786512017613366-32144-0/minicluster-data/master-0/data/info.pb
I20260812 05:20:42.999303 8959 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 05:20:43.001582 8959 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ee14c40d691b4a77b1c4067fdfaa5732: Bootstrap starting.
I20260812 05:20:43.002108 8959 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ee14c40d691b4a77b1c4067fdfaa5732: Neither blocks nor log segments found. Creating new log.
I20260812 05:20:43.002354 8959 log.cc:824] T 00000000000000000000000000000000 P ee14c40d691b4a77b1c4067fdfaa5732: Log is configured to *not* fsync() on all Append() calls
I20260812 05:20:43.002969 8959 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ee14c40d691b4a77b1c4067fdfaa5732: No bootstrap required, opened a new log
I20260812 05:20:43.004484 32144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0waFvT/build/release/bin/kudu as pid 8862
I20260812 05:20:43.004585 32144 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786512017613366-32144-0/minicluster-data/master-0/wal/instance
I20260812 05:20:43.004789 32144 external_mini_cluster.cc:1428] Setting key 5b7fced30d959b7bc6e029c01b75286d
I20260812 05:20:43.004671 8959 raft_consensus.cc:348] T 00000000000000000000000000000000 P ee14c40d691b4a77b1c4067fdfaa5732 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee14c40d691b4a77b1c4067fdfaa5732" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 34661 } }
I20260812 05:20:43.004827 8959 raft_consensus.cc:374] T 00000000000000000000000000000000 P ee14c40d691b4a77b1c4067fdfaa5732 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 05:20:43.004870 8959 raft_consensus.cc:729] T 00000000000000000000000000000000 P ee14c40d691b4a77b1c4067fdfaa5732 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee14c40d691b4a77b1c4067fdfaa5732, State: Initialized, Role: FOLLOWER
I20260812 05:20:43.005007 8959 consensus_queue.cc:260] T 00000000000000000000000000000000 P ee14c40d691b4a77b1c4067fdfaa5732 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee14c40d691b4a77b1c4067fdfaa5732" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 34661 } }
I20260812 05:20:43.005071 8959 raft_consensus.cc:388] T 00000000000000000000000000000000 P ee14c40d691b4a77b1c4067fdfaa5732 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 05:20:43.005091 8959 raft_consensus.cc:482] T 00000000000000000000000000000000 P ee14c40d691b4a77b1c4067fdfaa5732 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 05:20:43.005137 8959 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ee14c40d691b4a77b1c4067fdfaa5732 [term 0 FOLLOWER]: Advancing to term 1
I20260812 05:20:43.005757 8959 raft_consensus.cc:504] T 00000000000000000000000000000000 P ee14c40d691b4a77b1c4067fdfaa5732 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee14c40d691b4a77b1c4067fdfaa5732" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 34661 } }
I20260812 05:20:43.005856 8959 leader_election.cc:302] T 00000000000000000000000000000000 P ee14c40d691b4a77b1c4067fdfaa5732 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ee14c40d691b4a77b1c4067fdfaa5732; no voters:
I20260812 05:20:43.006013 8959 leader_election.cc:288] T 00000000000000000000000000000000 P ee14c40d691b4a77b1c4067fdfaa5732 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 05:20:43.006052 8966 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ee14c40d691b4a77b1c4067fdfaa5732 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 05:20:43.006201 8966 raft_consensus.cc:686] T 00000000000000000000000000000000 P ee14c40d691b4a77b1c4067fdfaa5732 [term 1 LEADER]: Becoming Leader. State: Replica: ee14c40d691b4a77b1c4067fdfaa5732, State: Running, Role: LEADER
I20260812 05:20:43.006245 8959 sys_catalog.cc:564] T 00000000000000000000000000000000 P ee14c40d691b4a77b1c4067fdfaa5732 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 05:20:43.006290 8966 consensus_queue.cc:237] T 00000000000000000000000000000000 P ee14c40d691b4a77b1c4067fdfaa5732 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee14c40d691b4a77b1c4067fdfaa5732" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 34661 } }
I20260812 05:20:43.006614 8966 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee14c40d691b4a77b1c4067fdfaa5732 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ee14c40d691b4a77b1c4067fdfaa5732. Latest consensus state: current_term: 1 leader_uuid: "ee14c40d691b4a77b1c4067fdfaa5732" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee14c40d691b4a77b1c4067fdfaa5732" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 34661 } } }
I20260812 05:20:43.006700 8966 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee14c40d691b4a77b1c4067fdfaa5732 [sys.catalog]: This master's current role is: LEADER
I20260812 05:20:43.006623 8968 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee14c40d691b4a77b1c4067fdfaa5732 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ee14c40d691b4a77b1c4067fdfaa5732" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee14c40d691b4a77b1c4067fdfaa5732" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 34661 } } }
I20260812 05:20:43.006773 8968 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee14c40d691b4a77b1c4067fdfaa5732 [sys.catalog]: This master's current role is: LEADER
I20260812 05:20:43.007458 8974 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 05:20:43.007941 8974 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 05:20:43.009289 8974 catalog_manager.cc:1347] Generated new cluster ID: be9e603b08c54d8280b26517092c7b63
I20260812 05:20:43.009341 8974 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 12 05:20:43 dist-test-slave-42z9 krb5kdc[8841](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512042, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.100.62@KRBTEST.COM
I20260812 05:20:43.015137 8965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:43.005421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37666 (local address 127.31.100.62:34661)
0812 05:20:43.005923 (+ 502us) server_negotiation.cc:206] Beginning negotiation
0812 05:20:43.005931 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 05:20:43.005965 (+ 34us) server_negotiation.cc:407] Connection header received
0812 05:20:43.006081 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:20:43.006089 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:20:43.006146 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:20:43.007123 (+ 977us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:20:43.007229 (+ 106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:43.008591 (+ 1362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:43.009523 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:43.009945 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:43.010041 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:43.012178 (+ 2137us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 05:20:43.012183 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 05:20:43.012183 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 05:20:43.012194 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0812 05:20:43.013766 (+ 1572us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:20:43.014082 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:20:43.014086 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:20:43.014086 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:20:43.014179 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:20:43.014351 (+ 172us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:20:43.014352 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:20:43.014352 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:20:43.014576 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 05:20:43.014719 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0812 05:20:43.014821 (+ 102us) server_negotiation.cc:299] Negotiation successful
0812 05:20:43.014953 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":442,"thread_start_us":389,"threads_started":1}
I20260812 05:20:43.028384 8974 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 05:20:43.028900 8974 catalog_manager.cc:1504] Loading token signing keys...
I20260812 05:20:43.031358 8974 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ee14c40d691b4a77b1c4067fdfaa5732: Generated new TSK 0
I20260812 05:20:43.031525 8974 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.31.100.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.100.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.100.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.1.keytab.
Entry for principal kudu/127.31.100.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.1.keytab.
I20260812 05:20:43.058168 32144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0waFvT/build/release/bin/kudu
/tmp/dist-test-task0waFvT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786512017613366-32144-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786512017613366-32144-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786512017613366-32144-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786512017613366-32144-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.100.1:0
--local_ip_for_outbound_sockets=127.31.100.1
--webserver_interface=127.31.100.1
--webserver_port=0
--tserver_master_addrs=127.31.100.62:34661
--keytab_file=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.1.keytab
--principal=kudu/127.31.100.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.100.20:40397
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786512017613366-32144-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786512017613366-32144-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786512017613366-32144-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 05:20:43.140028 9008 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 05:20:43.140237 9008 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 05:20:43.140261 9008 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 05:20:43.141489 9008 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 05:20:43.141531 9008 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 05:20:43.141557 9008 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.100.1
I20260812 05:20:43.143057 9008 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.100.20:40397
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786512017613366-32144-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786512017613366-32144-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.1.keytab
--principal=kudu/127.31.100.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.100.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786512017613366-32144-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.100.1
--webserver_port=0
--tserver_master_addrs=127.31.100.62:34661
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9008
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.100.1
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786512017613366-32144-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 04:20:34 UTC on ccycloud.quasar-phqdqp.root.comops.site
build id 4581871
I20260812 05:20:43.143293 9008 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 05:20:43.143530 9008 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 05:20:43.148430 9019 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 05:20:43.148530 9014 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 05:20:43.148435 9016 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 05:20:43.148763 9008 server_base.cc:1034] running on GCE node
I20260812 05:20:43.148972 9008 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 05:20:43.149159 9008 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 05:20:43.150312 9008 hybrid_clock.cc:648] HybridClock initialized: now 1786512043150294 us; error 29 us; skew 500 ppm
Aug 12 05:20:43 dist-test-slave-42z9 krb5kdc[8841](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512043, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.100.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 05:20:43.153620 9008 init.cc:377] Logged in from keytab as kudu/127.31.100.1@KRBTEST.COM (short username kudu)
I20260812 05:20:43.154170 9008 webserver.cc:460] Webserver started at http://127.31.100.1:44403/ using document root <none> and password file <none>
I20260812 05:20:43.154464 9008 fs_manager.cc:359] Metadata directory not provided
I20260812 05:20:43.154532 9008 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 05:20:43.154672 9008 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 05:20:43.156100 9008 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786512017613366-32144-0/minicluster-data/ts-0/data/instance:
uuid: "ee38cc01e7814a92a5496b9d863e6022"
format_stamp: "Formatted at 2026-08-12 05:20:43 on dist-test-slave-42z9"
server_key: "6ab1a4814ddfa67809ef623c2f169d64"
server_key_iv: "a2ad45cf578e5809df681d9fad0cf72a"
server_key_version: "encryptionkey@0"
I20260812 05:20:43.156467 9008 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786512017613366-32144-0/minicluster-data/ts-0/wal/instance:
uuid: "ee38cc01e7814a92a5496b9d863e6022"
format_stamp: "Formatted at 2026-08-12 05:20:43 on dist-test-slave-42z9"
server_key: "6ab1a4814ddfa67809ef623c2f169d64"
server_key_iv: "a2ad45cf578e5809df681d9fad0cf72a"
server_key_version: "encryptionkey@0"
I20260812 05:20:43.157713 9008 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260812 05:20:43.158411 9032 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 05:20:43.158566 9008 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260812 05:20:43.158653 9008 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786512017613366-32144-0/minicluster-data/ts-0/data,/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786512017613366-32144-0/minicluster-data/ts-0/wal
uuid: "ee38cc01e7814a92a5496b9d863e6022"
format_stamp: "Formatted at 2026-08-12 05:20:43 on dist-test-slave-42z9"
server_key: "6ab1a4814ddfa67809ef623c2f169d64"
server_key_iv: "a2ad45cf578e5809df681d9fad0cf72a"
server_key_version: "encryptionkey@0"
I20260812 05:20:43.158706 9008 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786512017613366-32144-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786512017613366-32144-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786512017613366-32144-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 05:20:43.181852 9008 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 05:20:43.183852 9008 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 05:20:43.183960 9008 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 05:20:43.184130 9008 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 05:20:43.184368 9008 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 05:20:43.184406 9008 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 05:20:43.184427 9008 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 05:20:43.184442 9008 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 05:20:43.189644 9008 rpc_server.cc:307] RPC server started. Bound to: 127.31.100.1:34253
I20260812 05:20:43.189708 9220 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.100.1:34253 every 8 connection(s)
I20260812 05:20:43.189986 9008 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786512017613366-32144-0/minicluster-data/ts-0/data/info.pb
I20260812 05:20:43.193608 32144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0waFvT/build/release/bin/kudu as pid 9008
I20260812 05:20:43.193717 32144 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786512017613366-32144-0/minicluster-data/ts-0/wal/instance
I20260812 05:20:43.193888 32144 external_mini_cluster.cc:1428] Setting key 409b8eab67f58c5223c54816053cb74e
Aug 12 05:20:43 dist-test-slave-42z9 krb5kdc[8841](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512043, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.100.1@KRBTEST.COM for kudu/127.31.100.62@KRBTEST.COM
I20260812 05:20:43.199208 8965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:43.190677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.100.1:60257 (local address 127.31.100.62:34661)
0812 05:20:43.190759 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0812 05:20:43.190760 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0812 05:20:43.190925 (+ 165us) server_negotiation.cc:407] Connection header received
0812 05:20:43.191847 (+ 922us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:20:43.191848 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:20:43.191864 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:20:43.191929 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:20:43.192499 (+ 570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:43.193011 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:43.194095 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:43.194298 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:43.194365 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:43.196606 (+ 2241us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 05:20:43.196607 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 05:20:43.196608 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 05:20:43.196616 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0812 05:20:43.197859 (+ 1243us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:20:43.198319 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:20:43.198320 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:20:43.198321 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:20:43.198384 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:20:43.198613 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:20:43.198614 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:20:43.198614 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:20:43.198738 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 05:20:43.198808 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0812 05:20:43.199032 (+ 224us) server_negotiation.cc:299] Negotiation successful
0812 05:20:43.199119 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260812 05:20:43.199404 9225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:43.190627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.62:34661 (local address 127.31.100.1:60257)
0812 05:20:43.190785 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0812 05:20:43.190826 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0812 05:20:43.191732 (+ 906us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:20:43.191990 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:20:43.191993 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:20:43.192229 (+ 236us) client_negotiation.cc:189] Negotiated authn=SASL
0812 05:20:43.192417 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:20:43.192421 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:20:43.193073 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:20:43.193074 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:20:43.193991 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:20:43.193995 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:20:43.194398 (+ 403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:20:43.194399 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:20:43.194534 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:43.194997 (+ 463us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 05:20:43.195004 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0812 05:20:43.196518 (+ 1514us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 05:20:43.198009 (+ 1491us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 05:20:43.198016 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 05:20:43.198018 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 05:20:43.198235 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 05:20:43.198468 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 05:20:43.198469 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 05:20:43.198469 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 05:20:43.198563 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 05:20:43.198804 (+ 241us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 05:20:43.198806 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 05:20:43.198908 (+ 102us) client_negotiation.cc:769] Sending connection context
0812 05:20:43.198985 (+ 77us) client_negotiation.cc:240] Negotiation successful
0812 05:20:43.199117 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":76,"threads_started":1}
I20260812 05:20:43.199865 9221 heartbeater.cc:347] Connected to a master server at 127.31.100.62:34661
I20260812 05:20:43.199951 9221 heartbeater.cc:464] Registering TS with master...
I20260812 05:20:43.200114 9221 heartbeater.cc:511] Master 127.31.100.62:34661 requested a full tablet report, sending...
I20260812 05:20:43.200592 8910 ts_manager.cc:194] Registered new tserver with Master: ee38cc01e7814a92a5496b9d863e6022 (127.31.100.1:34253)
I20260812 05:20:43.201201 8910 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.100.1@KRBTEST.COM'} at 127.31.100.1:60257
WARNING: no policy specified for kudu/127.31.100.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.100.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.100.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.2.keytab.
Entry for principal kudu/127.31.100.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.2.keytab.
I20260812 05:20:43.220114 32144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0waFvT/build/release/bin/kudu
/tmp/dist-test-task0waFvT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786512017613366-32144-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786512017613366-32144-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786512017613366-32144-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786512017613366-32144-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.100.2:0
--local_ip_for_outbound_sockets=127.31.100.2
--webserver_interface=127.31.100.2
--webserver_port=0
--tserver_master_addrs=127.31.100.62:34661
--keytab_file=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.2.keytab
--principal=kudu/127.31.100.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.100.20:40397
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786512017613366-32144-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786512017613366-32144-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786512017613366-32144-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 05:20:43.299096 9230 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 05:20:43.299284 9230 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 05:20:43.299315 9230 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 05:20:43.300421 9230 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 05:20:43.300462 9230 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 05:20:43.300496 9230 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.100.2
I20260812 05:20:43.301924 9230 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.100.20:40397
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786512017613366-32144-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786512017613366-32144-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.2.keytab
--principal=kudu/127.31.100.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.100.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786512017613366-32144-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.100.2
--webserver_port=0
--tserver_master_addrs=127.31.100.62:34661
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9230
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.100.2
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786512017613366-32144-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 04:20:34 UTC on ccycloud.quasar-phqdqp.root.comops.site
build id 4581871
I20260812 05:20:43.302161 9230 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 05:20:43.302351 9230 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 05:20:43.306967 9237 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 05:20:43.307099 9230 server_base.cc:1034] running on GCE node
W20260812 05:20:43.307099 9238 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 05:20:43.307022 9241 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 05:20:43.307380 9230 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 05:20:43.307633 9230 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 05:20:43.308771 9230 hybrid_clock.cc:648] HybridClock initialized: now 1786512043308761 us; error 32 us; skew 500 ppm
Aug 12 05:20:43 dist-test-slave-42z9 krb5kdc[8841](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512043, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.100.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 05:20:43.311182 9230 init.cc:377] Logged in from keytab as kudu/127.31.100.2@KRBTEST.COM (short username kudu)
I20260812 05:20:43.311590 9230 webserver.cc:460] Webserver started at http://127.31.100.2:33161/ using document root <none> and password file <none>
I20260812 05:20:43.311779 9230 fs_manager.cc:359] Metadata directory not provided
I20260812 05:20:43.311826 9230 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 05:20:43.311949 9230 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 05:20:43.312896 9230 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786512017613366-32144-0/minicluster-data/ts-1/data/instance:
uuid: "ff7b1a598cd4473ea1d5897d70a00975"
format_stamp: "Formatted at 2026-08-12 05:20:43 on dist-test-slave-42z9"
server_key: "27bd1d63dd08afa3a844fa1bd7ef4306"
server_key_iv: "a0fc55caaba22114510cceb7a7c5efa3"
server_key_version: "encryptionkey@0"
I20260812 05:20:43.313215 9230 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786512017613366-32144-0/minicluster-data/ts-1/wal/instance:
uuid: "ff7b1a598cd4473ea1d5897d70a00975"
format_stamp: "Formatted at 2026-08-12 05:20:43 on dist-test-slave-42z9"
server_key: "27bd1d63dd08afa3a844fa1bd7ef4306"
server_key_iv: "a0fc55caaba22114510cceb7a7c5efa3"
server_key_version: "encryptionkey@0"
I20260812 05:20:43.314579 9230 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260812 05:20:43.315229 9250 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 05:20:43.315390 9230 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260812 05:20:43.315493 9230 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786512017613366-32144-0/minicluster-data/ts-1/data,/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786512017613366-32144-0/minicluster-data/ts-1/wal
uuid: "ff7b1a598cd4473ea1d5897d70a00975"
format_stamp: "Formatted at 2026-08-12 05:20:43 on dist-test-slave-42z9"
server_key: "27bd1d63dd08afa3a844fa1bd7ef4306"
server_key_iv: "a0fc55caaba22114510cceb7a7c5efa3"
server_key_version: "encryptionkey@0"
I20260812 05:20:43.315553 9230 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786512017613366-32144-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786512017613366-32144-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786512017613366-32144-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 05:20:43.334195 9230 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 05:20:43.336182 9230 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 05:20:43.336313 9230 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 05:20:43.336519 9230 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 05:20:43.336794 9230 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 05:20:43.336826 9230 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 05:20:43.336853 9230 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 05:20:43.336870 9230 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 05:20:43.342332 9230 rpc_server.cc:307] RPC server started. Bound to: 127.31.100.2:45529
I20260812 05:20:43.342393 9425 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.100.2:45529 every 8 connection(s)
I20260812 05:20:43.342658 9230 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786512017613366-32144-0/minicluster-data/ts-1/data/info.pb
I20260812 05:20:43.344478 32144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0waFvT/build/release/bin/kudu as pid 9230
I20260812 05:20:43.344596 32144 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786512017613366-32144-0/minicluster-data/ts-1/wal/instance
I20260812 05:20:43.344806 32144 external_mini_cluster.cc:1428] Setting key 0d973749f7228589826ed031fdc5692c
Aug 12 05:20:43 dist-test-slave-42z9 krb5kdc[8841](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512043, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.100.2@KRBTEST.COM for kudu/127.31.100.62@KRBTEST.COM
I20260812 05:20:43.352191 8965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:43.343286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.100.2:46265 (local address 127.31.100.62:34661)
0812 05:20:43.343340 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0812 05:20:43.343342 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 05:20:43.343516 (+ 174us) server_negotiation.cc:407] Connection header received
0812 05:20:43.344423 (+ 907us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:20:43.344424 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:20:43.344435 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:20:43.344484 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:20:43.345253 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:43.345865 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:43.347131 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:43.347408 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:43.347471 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:43.349637 (+ 2166us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 05:20:43.349639 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 05:20:43.349639 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 05:20:43.349646 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 05:20:43.350888 (+ 1242us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:20:43.351342 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:20:43.351343 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:20:43.351343 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:20:43.351388 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:20:43.351636 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:20:43.351636 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:20:43.351636 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:20:43.351728 (+ 92us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 05:20:43.351769 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0812 05:20:43.352068 (+ 299us) server_negotiation.cc:299] Negotiation successful
0812 05:20:43.352128 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260812 05:20:43.352428 9428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:43.343252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.62:34661 (local address 127.31.100.2:46265)
0812 05:20:43.343393 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0812 05:20:43.343426 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0812 05:20:43.344315 (+ 889us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:20:43.344593 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:20:43.344597 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:20:43.344923 (+ 326us) client_negotiation.cc:189] Negotiated authn=SASL
0812 05:20:43.345168 (+ 245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:20:43.345174 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:20:43.345948 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:20:43.345949 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:20:43.347047 (+ 1098us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:20:43.347051 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:20:43.347496 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:20:43.347497 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:20:43.347597 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:43.347999 (+ 402us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 05:20:43.348007 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0812 05:20:43.349544 (+ 1537us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 05:20:43.350973 (+ 1429us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 05:20:43.350980 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 05:20:43.350982 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 05:20:43.351274 (+ 292us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 05:20:43.351442 (+ 168us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 05:20:43.351442 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 05:20:43.351443 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 05:20:43.351577 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 05:20:43.351778 (+ 201us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 05:20:43.351783 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 05:20:43.351936 (+ 153us) client_negotiation.cc:769] Sending connection context
0812 05:20:43.352058 (+ 122us) client_negotiation.cc:240] Negotiation successful
0812 05:20:43.352214 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":72,"threads_started":1}
I20260812 05:20:43.352902 9426 heartbeater.cc:347] Connected to a master server at 127.31.100.62:34661
I20260812 05:20:43.353006 9426 heartbeater.cc:464] Registering TS with master...
I20260812 05:20:43.353189 9426 heartbeater.cc:511] Master 127.31.100.62:34661 requested a full tablet report, sending...
I20260812 05:20:43.353569 8910 ts_manager.cc:194] Registered new tserver with Master: ff7b1a598cd4473ea1d5897d70a00975 (127.31.100.2:45529)
WARNING: no policy specified for kudu/127.31.100.3@KRBTEST.COM; defaulting to no policy
I20260812 05:20:43.353932 8910 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.100.2@KRBTEST.COM'} at 127.31.100.2:46265
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.100.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.100.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.3.keytab.
Entry for principal kudu/127.31.100.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.3.keytab.
I20260812 05:20:43.369604 32144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0waFvT/build/release/bin/kudu
/tmp/dist-test-task0waFvT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786512017613366-32144-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786512017613366-32144-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786512017613366-32144-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786512017613366-32144-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.100.3:0
--local_ip_for_outbound_sockets=127.31.100.3
--webserver_interface=127.31.100.3
--webserver_port=0
--tserver_master_addrs=127.31.100.62:34661
--keytab_file=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.3.keytab
--principal=kudu/127.31.100.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.100.20:40397
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786512017613366-32144-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786512017613366-32144-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786512017613366-32144-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 05:20:43.448810 9432 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 05:20:43.448998 9432 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 05:20:43.449029 9432 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 05:20:43.450193 9432 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 05:20:43.450240 9432 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 05:20:43.450275 9432 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.100.3
I20260812 05:20:43.451721 9432 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.100.20:40397
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786512017613366-32144-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786512017613366-32144-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.3.keytab
--principal=kudu/127.31.100.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.100.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786512017613366-32144-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.100.3
--webserver_port=0
--tserver_master_addrs=127.31.100.62:34661
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9432
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.100.3
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786512017613366-32144-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 04:20:34 UTC on ccycloud.quasar-phqdqp.root.comops.site
build id 4581871
I20260812 05:20:43.451951 9432 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 05:20:43.452147 9432 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 05:20:43.456753 9443 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 05:20:43.456770 9451 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 05:20:43.456763 9444 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 05:20:43.457053 9432 server_base.cc:1034] running on GCE node
I20260812 05:20:43.457240 9432 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 05:20:43.457485 9432 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 05:20:43.458629 9432 hybrid_clock.cc:648] HybridClock initialized: now 1786512043458617 us; error 29 us; skew 500 ppm
Aug 12 05:20:43 dist-test-slave-42z9 krb5kdc[8841](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512043, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.100.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 05:20:43.460935 9432 init.cc:377] Logged in from keytab as kudu/127.31.100.3@KRBTEST.COM (short username kudu)
I20260812 05:20:43.461354 9432 webserver.cc:460] Webserver started at http://127.31.100.3:34879/ using document root <none> and password file <none>
I20260812 05:20:43.461604 9432 fs_manager.cc:359] Metadata directory not provided
I20260812 05:20:43.461650 9432 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 05:20:43.461737 9432 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 05:20:43.462680 9432 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786512017613366-32144-0/minicluster-data/ts-2/data/instance:
uuid: "08820be739554cce870f665116d18a0f"
format_stamp: "Formatted at 2026-08-12 05:20:43 on dist-test-slave-42z9"
server_key: "4599bb2e7c9a873fa488a0c6cffc41b0"
server_key_iv: "2e1fc32a1028aba524f071847b25737b"
server_key_version: "encryptionkey@0"
I20260812 05:20:43.462980 9432 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786512017613366-32144-0/minicluster-data/ts-2/wal/instance:
uuid: "08820be739554cce870f665116d18a0f"
format_stamp: "Formatted at 2026-08-12 05:20:43 on dist-test-slave-42z9"
server_key: "4599bb2e7c9a873fa488a0c6cffc41b0"
server_key_iv: "2e1fc32a1028aba524f071847b25737b"
server_key_version: "encryptionkey@0"
I20260812 05:20:43.464171 9432 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260812 05:20:43.464794 9464 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 05:20:43.464931 9432 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260812 05:20:43.465009 9432 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786512017613366-32144-0/minicluster-data/ts-2/data,/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786512017613366-32144-0/minicluster-data/ts-2/wal
uuid: "08820be739554cce870f665116d18a0f"
format_stamp: "Formatted at 2026-08-12 05:20:43 on dist-test-slave-42z9"
server_key: "4599bb2e7c9a873fa488a0c6cffc41b0"
server_key_iv: "2e1fc32a1028aba524f071847b25737b"
server_key_version: "encryptionkey@0"
I20260812 05:20:43.465076 9432 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786512017613366-32144-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786512017613366-32144-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786512017613366-32144-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 05:20:43.484225 9432 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 05:20:43.486161 9432 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 05:20:43.486277 9432 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 05:20:43.486480 9432 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 05:20:43.486758 9432 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 05:20:43.486791 9432 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 05:20:43.486814 9432 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 05:20:43.486837 9432 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 05:20:43.492130 9432 rpc_server.cc:307] RPC server started. Bound to: 127.31.100.3:45643
I20260812 05:20:43.492165 9633 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.100.3:45643 every 8 connection(s)
I20260812 05:20:43.492446 9432 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786512017613366-32144-0/minicluster-data/ts-2/data/info.pb
I20260812 05:20:43.493898 32144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0waFvT/build/release/bin/kudu as pid 9432
I20260812 05:20:43.494009 32144 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786512017613366-32144-0/minicluster-data/ts-2/wal/instance
I20260812 05:20:43.494216 32144 external_mini_cluster.cc:1428] Setting key 6fb3910456b0ad158ea28aece5d66b9a
Aug 12 05:20:43 dist-test-slave-42z9 krb5kdc[8841](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512043, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.100.3@KRBTEST.COM for kudu/127.31.100.62@KRBTEST.COM
I20260812 05:20:43.500964 8965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:43.493117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.100.3:38399 (local address 127.31.100.62:34661)
0812 05:20:43.493200 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0812 05:20:43.493202 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 05:20:43.493345 (+ 143us) server_negotiation.cc:407] Connection header received
0812 05:20:43.494397 (+ 1052us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:20:43.494398 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:20:43.494412 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:20:43.494482 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:20:43.495216 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:43.495707 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:43.496621 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:43.496804 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:43.496849 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:43.498769 (+ 1920us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 05:20:43.498771 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 05:20:43.498771 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 05:20:43.498778 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 05:20:43.499916 (+ 1138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:20:43.500256 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:20:43.500257 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:20:43.500257 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:20:43.500308 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:20:43.500508 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:20:43.500508 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:20:43.500509 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:20:43.500600 (+ 91us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 05:20:43.500645 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0812 05:20:43.500834 (+ 189us) server_negotiation.cc:299] Negotiation successful
0812 05:20:43.500889 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260812 05:20:43.501122 9636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:43.493081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.62:34661 (local address 127.31.100.3:38399)
0812 05:20:43.493229 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0812 05:20:43.493267 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0812 05:20:43.494291 (+ 1024us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:20:43.494549 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:20:43.494554 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:20:43.494901 (+ 347us) client_negotiation.cc:189] Negotiated authn=SASL
0812 05:20:43.495151 (+ 250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:20:43.495157 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:20:43.495773 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:20:43.495774 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:20:43.496554 (+ 780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:20:43.496556 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:20:43.496858 (+ 302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:20:43.496859 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:20:43.496959 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:43.497318 (+ 359us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 05:20:43.497325 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0812 05:20:43.498692 (+ 1367us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 05:20:43.499991 (+ 1299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 05:20:43.499997 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 05:20:43.499999 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 05:20:43.500201 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 05:20:43.500351 (+ 150us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 05:20:43.500351 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 05:20:43.500351 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 05:20:43.500463 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 05:20:43.500642 (+ 179us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 05:20:43.500644 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 05:20:43.500743 (+ 99us) client_negotiation.cc:769] Sending connection context
0812 05:20:43.500816 (+ 73us) client_negotiation.cc:240] Negotiation successful
0812 05:20:43.500926 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":77,"threads_started":1}
I20260812 05:20:43.501596 9634 heartbeater.cc:347] Connected to a master server at 127.31.100.62:34661
I20260812 05:20:43.501683 9634 heartbeater.cc:464] Registering TS with master...
I20260812 05:20:43.501838 9634 heartbeater.cc:511] Master 127.31.100.62:34661 requested a full tablet report, sending...
I20260812 05:20:43.502158 8910 ts_manager.cc:194] Registered new tserver with Master: 08820be739554cce870f665116d18a0f (127.31.100.3:45643)
I20260812 05:20:43.502486 8910 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.100.3@KRBTEST.COM'} at 127.31.100.3:38399
I20260812 05:20:43.505870 32144 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786512017613366-32144-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786512017613366-32144-0/krb5kdc/test-admin.keytab.
Aug 12 05:20:43 dist-test-slave-42z9 krb5kdc[8841](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512043, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 05:20:43.520886 32144 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260812 05:20:43.522444 9644 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 05:20:43.521696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50958 (local address 127.31.100.1:34253)
0812 05:20:43.521853 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0812 05:20:43.521857 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 05:20:43.521866 (+ 9us) server_negotiation.cc:407] Connection header received
0812 05:20:43.521889 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:20:43.521891 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:20:43.521904 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:20:43.521972 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:20:43.522353 (+ 381us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50958: BlockingRecv error: recv got EOF from 127.0.0.1:50958 (error 108)
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":71,"threads_started":1}
W20260812 05:20:43.522617 32144 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786512017613366-32144-0/krb5kdc/krb5cc' of type FILE
Aug 12 05:20:43 dist-test-slave-42z9 krb5kdc[8841](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512043, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 05:20:43.523562 32144 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 12 05:20:43 dist-test-slave-42z9 krb5kdc[8841](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512043, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 05:20:43.524384 32144 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 12 05:20:43 dist-test-slave-42z9 krb5kdc[8841](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512043, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.100.1@KRBTEST.COM
I20260812 05:20:43.531268 9644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:43.524831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50966 (local address 127.31.100.1:34253)
0812 05:20:43.524881 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0812 05:20:43.524883 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 05:20:43.524941 (+ 58us) server_negotiation.cc:407] Connection header received
0812 05:20:43.524985 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:20:43.524987 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:20:43.524996 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:20:43.525039 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:20:43.525357 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:43.526090 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:43.526744 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:43.527042 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:43.527091 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:43.528908 (+ 1817us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 05:20:43.528916 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 05:20:43.528916 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 05:20:43.528943 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0812 05:20:43.530310 (+ 1367us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:20:43.530605 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:20:43.530607 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:20:43.530608 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:20:43.530661 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:20:43.530836 (+ 175us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:20:43.530837 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:20:43.530837 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:20:43.530950 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 05:20:43.531037 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0812 05:20:43.531097 (+ 60us) server_negotiation.cc:299] Negotiation successful
0812 05:20:43.531153 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
W20260812 05:20:43.532766 9644 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 05:20:43.532367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50978 (local address 127.31.100.1:34253)
0812 05:20:43.532426 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0812 05:20:43.532428 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 05:20:43.532437 (+ 9us) server_negotiation.cc:407] Connection header received
0812 05:20:43.532491 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:20:43.532492 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:20:43.532499 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:20:43.532538 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:20:43.532735 (+ 197us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50978: BlockingRecv error: recv got EOF from 127.0.0.1:50978 (error 108)
Metrics: {"server-negotiator.queue_time_us":15}
W20260812 05:20:43.532979 32144 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786512017613366-32144-0/krb5kdc/krb5cc' of type FILE
Aug 12 05:20:43 dist-test-slave-42z9 krb5kdc[8841](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512043, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 05:20:43.533828 32144 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 12 05:20:43 dist-test-slave-42z9 krb5kdc[8841](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512043, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 05:20:43.534605 32144 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 12 05:20:43 dist-test-slave-42z9 krb5kdc[8841](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512043, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.100.1@KRBTEST.COM
I20260812 05:20:43.540414 9644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:43.535035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50984 (local address 127.31.100.1:34253)
0812 05:20:43.535071 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0812 05:20:43.535072 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0812 05:20:43.535116 (+ 44us) server_negotiation.cc:407] Connection header received
0812 05:20:43.535151 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:20:43.535152 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:20:43.535160 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:20:43.535199 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:20:43.535498 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:43.535928 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:43.536547 (+ 619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:43.536732 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:43.536782 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:43.538503 (+ 1721us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 05:20:43.538504 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 05:20:43.538504 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 05:20:43.538510 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0812 05:20:43.539641 (+ 1131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:20:43.539904 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:20:43.539905 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:20:43.539905 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:20:43.539950 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:20:43.540086 (+ 136us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:20:43.540086 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:20:43.540086 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:20:43.540157 (+ 71us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 05:20:43.540214 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0812 05:20:43.540288 (+ 74us) server_negotiation.cc:299] Negotiation successful
0812 05:20:43.540341 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
W20260812 05:20:43.541765 9644 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 05:20:43.541295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51000 (local address 127.31.100.1:34253)
0812 05:20:43.541341 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0812 05:20:43.541343 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 05:20:43.541396 (+ 53us) server_negotiation.cc:407] Connection header received
0812 05:20:43.541434 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:20:43.541434 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:20:43.541461 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:20:43.541510 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:20:43.541723 (+ 213us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51000: BlockingRecv error: recv got EOF from 127.0.0.1:51000 (error 108)
Metrics: {"server-negotiator.queue_time_us":29}
W20260812 05:20:43.541848 32144 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786512017613366-32144-0/krb5kdc/krb5cc' of type FILE
Aug 12 05:20:43 dist-test-slave-42z9 krb5kdc[8841](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512043, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 05:20:43.542665 32144 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 12 05:20:43 dist-test-slave-42z9 krb5kdc[8841](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512043, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 05:20:43.543385 32144 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 12 05:20:43 dist-test-slave-42z9 krb5kdc[8841](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512043, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.100.1@KRBTEST.COM
I20260812 05:20:43.548947 9644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:43.543802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51016 (local address 127.31.100.1:34253)
0812 05:20:43.543846 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0812 05:20:43.543847 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0812 05:20:43.543856 (+ 9us) server_negotiation.cc:407] Connection header received
0812 05:20:43.543919 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:20:43.543920 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:20:43.543926 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:20:43.543966 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:20:43.544283 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:43.544725 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:43.545310 (+ 585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:43.545551 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:43.545602 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:43.547216 (+ 1614us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 05:20:43.547217 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 05:20:43.547217 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 05:20:43.547223 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0812 05:20:43.548211 (+ 988us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:20:43.548476 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:20:43.548477 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:20:43.548477 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:20:43.548516 (+ 39us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:20:43.548631 (+ 115us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:20:43.548631 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:20:43.548631 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:20:43.548699 (+ 68us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 05:20:43.548755 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0812 05:20:43.548815 (+ 60us) server_negotiation.cc:299] Negotiation successful
0812 05:20:43.548875 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260812 05:20:43.549573 32144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0waFvT/build/release/bin/kudu with pid 9008
I20260812 05:20:43.565513 32144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0waFvT/build/release/bin/kudu with pid 9230
I20260812 05:20:43.576301 32144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0waFvT/build/release/bin/kudu with pid 9432
I20260812 05:20:43.585424 32144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0waFvT/build/release/bin/kudu with pid 8862
2026-08-12T05:20:43Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (1990 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786512017613366-32144-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 05:20:43 dist-test-slave-42z9 krb5kdc[9665](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 05:20:43 dist-test-slave-42z9 krb5kdc[9665](info): set up 2 sockets
Aug 12 05:20:43 dist-test-slave-42z9 krb5kdc[9665](info): commencing operation
krb5kdc: starting...
W20260812 05:20:44.751593 32144 mini_kdc.cc:121] Time spent starting KDC: real 1.142s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786512017613366-32144-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786512017613366-32144-0/krb5kdc/test-user.keytab.
Aug 12 05:20:44 dist-test-slave-42z9 krb5kdc[9665](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512044, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T05:20:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T05:20:44Z Disabled control of system clock
WARNING: no policy specified for oryx/127.31.100.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.31.100.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.31.100.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786512017613366-32144-0/krb5kdc/oryx_127.31.100.62.keytab.
Entry for principal oryx/127.31.100.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786512017613366-32144-0/krb5kdc/oryx_127.31.100.62.keytab.
I20260812 05:20:44.863510 32144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0waFvT/build/release/bin/kudu
/tmp/dist-test-task0waFvT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786512017613366-32144-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786512017613366-32144-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786512017613366-32144-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786512017613366-32144-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.100.62:35003
--webserver_interface=127.31.100.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786512017613366-32144-0/krb5kdc/oryx_127.31.100.62.keytab
--principal=oryx/127.31.100.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.100.20:44403
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.100.62:35003
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786512017613366-32144-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786512017613366-32144-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786512017613366-32144-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 05:20:44.943354 9687 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 05:20:44.943557 9687 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 05:20:44.943578 9687 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 05:20:44.944707 9687 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 05:20:44.944748 9687 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 05:20:44.944761 9687 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 05:20:44.944772 9687 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 05:20:44.944782 9687 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 05:20:44.946269 9687 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.100.20:44403
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786512017613366-32144-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786512017613366-32144-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.100.62:35003
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786512017613366-32144-0/krb5kdc/oryx_127.31.100.62.keytab
--principal=oryx/127.31.100.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.100.62:35003
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786512017613366-32144-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.100.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9687
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786512017613366-32144-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 04:20:34 UTC on ccycloud.quasar-phqdqp.root.comops.site
build id 4581871
I20260812 05:20:44.946485 9687 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 05:20:44.946696 9687 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 05:20:44.951279 9700 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 05:20:44.951320 9705 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 05:20:44.951264 9703 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 05:20:44.951444 9687 server_base.cc:1034] running on GCE node
I20260812 05:20:44.951788 9687 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 05:20:44.952080 9687 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 05:20:44.953233 9687 hybrid_clock.cc:648] HybridClock initialized: now 1786512044953215 us; error 28 us; skew 500 ppm
Aug 12 05:20:44 dist-test-slave-42z9 krb5kdc[9665](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512044, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.100.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 05:20:44.959842 9687 init.cc:377] Logged in from keytab as oryx/127.31.100.62@KRBTEST.COM (short username oryx)
I20260812 05:20:44.960263 9687 webserver.cc:460] Webserver started at http://127.31.100.62:39509/ using document root <none> and password file <none>
I20260812 05:20:44.960472 9687 fs_manager.cc:359] Metadata directory not provided
I20260812 05:20:44.960521 9687 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 05:20:44.960654 9687 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 05:20:44.961774 9687 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786512017613366-32144-0/minicluster-data/master-0/data/instance:
uuid: "070ed78c702d4333919d5d31da7b7608"
format_stamp: "Formatted at 2026-08-12 05:20:44 on dist-test-slave-42z9"
server_key: "50912eb46b6d5bbedf6cd950f00fe1b7"
server_key_iv: "c7f8e1e1cb724777757b50d3fa9259eb"
server_key_version: "encryptionkey@0"
I20260812 05:20:44.962100 9687 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786512017613366-32144-0/minicluster-data/master-0/wal/instance:
uuid: "070ed78c702d4333919d5d31da7b7608"
format_stamp: "Formatted at 2026-08-12 05:20:44 on dist-test-slave-42z9"
server_key: "50912eb46b6d5bbedf6cd950f00fe1b7"
server_key_iv: "c7f8e1e1cb724777757b50d3fa9259eb"
server_key_version: "encryptionkey@0"
I20260812 05:20:44.963390 9687 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260812 05:20:44.964164 9716 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 05:20:44.964318 9687 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260812 05:20:44.964411 9687 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786512017613366-32144-0/minicluster-data/master-0/data,/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786512017613366-32144-0/minicluster-data/master-0/wal
uuid: "070ed78c702d4333919d5d31da7b7608"
format_stamp: "Formatted at 2026-08-12 05:20:44 on dist-test-slave-42z9"
server_key: "50912eb46b6d5bbedf6cd950f00fe1b7"
server_key_iv: "c7f8e1e1cb724777757b50d3fa9259eb"
server_key_version: "encryptionkey@0"
I20260812 05:20:44.964494 9687 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786512017613366-32144-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786512017613366-32144-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786512017613366-32144-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 05:20:44.984934 9687 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 05:20:44.987082 9687 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 05:20:44.987221 9687 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 05:20:44.991370 9687 rpc_server.cc:307] RPC server started. Bound to: 127.31.100.62:35003
I20260812 05:20:44.991499 9795 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.100.62:35003 every 8 connection(s)
I20260812 05:20:44.991771 9687 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786512017613366-32144-0/minicluster-data/master-0/data/info.pb
I20260812 05:20:44.992885 9797 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 05:20:44.995322 9797 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 070ed78c702d4333919d5d31da7b7608: Bootstrap starting.
I20260812 05:20:44.995886 9797 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 070ed78c702d4333919d5d31da7b7608: Neither blocks nor log segments found. Creating new log.
I20260812 05:20:44.996160 9797 log.cc:824] T 00000000000000000000000000000000 P 070ed78c702d4333919d5d31da7b7608: Log is configured to *not* fsync() on all Append() calls
I20260812 05:20:44.996851 9797 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 070ed78c702d4333919d5d31da7b7608: No bootstrap required, opened a new log
I20260812 05:20:44.998245 32144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0waFvT/build/release/bin/kudu as pid 9687
I20260812 05:20:44.998355 32144 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786512017613366-32144-0/minicluster-data/master-0/wal/instance
I20260812 05:20:44.998569 32144 external_mini_cluster.cc:1428] Setting key 7abb049e41477194f546f37ada25cb9d
I20260812 05:20:44.998553 9797 raft_consensus.cc:348] T 00000000000000000000000000000000 P 070ed78c702d4333919d5d31da7b7608 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "070ed78c702d4333919d5d31da7b7608" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 35003 } }
I20260812 05:20:44.998703 9797 raft_consensus.cc:374] T 00000000000000000000000000000000 P 070ed78c702d4333919d5d31da7b7608 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 05:20:44.998749 9797 raft_consensus.cc:729] T 00000000000000000000000000000000 P 070ed78c702d4333919d5d31da7b7608 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 070ed78c702d4333919d5d31da7b7608, State: Initialized, Role: FOLLOWER
I20260812 05:20:44.998893 9797 consensus_queue.cc:260] T 00000000000000000000000000000000 P 070ed78c702d4333919d5d31da7b7608 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "070ed78c702d4333919d5d31da7b7608" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 35003 } }
I20260812 05:20:44.998979 9797 raft_consensus.cc:388] T 00000000000000000000000000000000 P 070ed78c702d4333919d5d31da7b7608 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 05:20:44.999022 9797 raft_consensus.cc:482] T 00000000000000000000000000000000 P 070ed78c702d4333919d5d31da7b7608 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 05:20:44.999063 9797 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 070ed78c702d4333919d5d31da7b7608 [term 0 FOLLOWER]: Advancing to term 1
I20260812 05:20:44.999727 9797 raft_consensus.cc:504] T 00000000000000000000000000000000 P 070ed78c702d4333919d5d31da7b7608 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "070ed78c702d4333919d5d31da7b7608" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 35003 } }
I20260812 05:20:44.999832 9797 leader_election.cc:302] T 00000000000000000000000000000000 P 070ed78c702d4333919d5d31da7b7608 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 070ed78c702d4333919d5d31da7b7608; no voters:
I20260812 05:20:44.999984 9797 leader_election.cc:288] T 00000000000000000000000000000000 P 070ed78c702d4333919d5d31da7b7608 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 05:20:45.000020 9803 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 070ed78c702d4333919d5d31da7b7608 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 05:20:45.000154 9803 raft_consensus.cc:686] T 00000000000000000000000000000000 P 070ed78c702d4333919d5d31da7b7608 [term 1 LEADER]: Becoming Leader. State: Replica: 070ed78c702d4333919d5d31da7b7608, State: Running, Role: LEADER
I20260812 05:20:45.000245 9797 sys_catalog.cc:564] T 00000000000000000000000000000000 P 070ed78c702d4333919d5d31da7b7608 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 05:20:45.000273 9803 consensus_queue.cc:237] T 00000000000000000000000000000000 P 070ed78c702d4333919d5d31da7b7608 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "070ed78c702d4333919d5d31da7b7608" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 35003 } }
I20260812 05:20:45.000617 9804 sys_catalog.cc:455] T 00000000000000000000000000000000 P 070ed78c702d4333919d5d31da7b7608 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "070ed78c702d4333919d5d31da7b7608" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "070ed78c702d4333919d5d31da7b7608" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 35003 } } }
I20260812 05:20:45.000702 9804 sys_catalog.cc:458] T 00000000000000000000000000000000 P 070ed78c702d4333919d5d31da7b7608 [sys.catalog]: This master's current role is: LEADER
I20260812 05:20:45.000870 9805 sys_catalog.cc:455] T 00000000000000000000000000000000 P 070ed78c702d4333919d5d31da7b7608 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 070ed78c702d4333919d5d31da7b7608. Latest consensus state: current_term: 1 leader_uuid: "070ed78c702d4333919d5d31da7b7608" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "070ed78c702d4333919d5d31da7b7608" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 35003 } } }
I20260812 05:20:45.000957 9805 sys_catalog.cc:458] T 00000000000000000000000000000000 P 070ed78c702d4333919d5d31da7b7608 [sys.catalog]: This master's current role is: LEADER
I20260812 05:20:45.001104 9813 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 05:20:45.001653 9813 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 05:20:45.003317 9813 catalog_manager.cc:1347] Generated new cluster ID: 1c12969b4cd344eab17b8b9fb1e66566
I20260812 05:20:45.003368 9813 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 12 05:20:45 dist-test-slave-42z9 krb5kdc[9665](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512044, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.31.100.62@KRBTEST.COM
I20260812 05:20:45.009580 9802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:44.999027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48160 (local address 127.31.100.62:35003)
0812 05:20:44.999242 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0812 05:20:44.999251 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 05:20:44.999285 (+ 34us) server_negotiation.cc:407] Connection header received
0812 05:20:44.999703 (+ 418us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:20:44.999713 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:20:44.999777 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:20:44.999926 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:20:45.000756 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:45.002167 (+ 1411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:45.003177 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:45.003672 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:45.003771 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:45.005979 (+ 2208us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 05:20:45.005983 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 05:20:45.005983 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 05:20:45.005995 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0812 05:20:45.008104 (+ 2109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:20:45.008447 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:20:45.008450 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:20:45.008450 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:20:45.008547 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:20:45.008726 (+ 179us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:20:45.008727 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:20:45.008727 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:20:45.008965 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 05:20:45.009128 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0812 05:20:45.009226 (+ 98us) server_negotiation.cc:299] Negotiation successful
0812 05:20:45.009359 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":165,"thread_start_us":93,"threads_started":1}
I20260812 05:20:45.014670 9813 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 05:20:45.015331 9813 catalog_manager.cc:1504] Loading token signing keys...
I20260812 05:20:45.019289 9813 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 070ed78c702d4333919d5d31da7b7608: Generated new TSK 0
I20260812 05:20:45.019469 9813 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.31.100.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.31.100.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.31.100.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786512017613366-32144-0/krb5kdc/oryx_127.31.100.1.keytab.
Entry for principal oryx/127.31.100.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786512017613366-32144-0/krb5kdc/oryx_127.31.100.1.keytab.
I20260812 05:20:45.047828 32144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0waFvT/build/release/bin/kudu
/tmp/dist-test-task0waFvT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786512017613366-32144-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786512017613366-32144-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786512017613366-32144-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786512017613366-32144-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.100.1:0
--local_ip_for_outbound_sockets=127.31.100.1
--webserver_interface=127.31.100.1
--webserver_port=0
--tserver_master_addrs=127.31.100.62:35003
--keytab_file=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786512017613366-32144-0/krb5kdc/oryx_127.31.100.1.keytab
--principal=oryx/127.31.100.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.100.20:44403
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786512017613366-32144-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786512017613366-32144-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786512017613366-32144-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 05:20:45.128386 9830 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 05:20:45.128576 9830 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 05:20:45.128598 9830 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 05:20:45.129670 9830 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 05:20:45.129701 9830 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 05:20:45.129725 9830 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.100.1
I20260812 05:20:45.131075 9830 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.100.20:44403
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786512017613366-32144-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786512017613366-32144-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786512017613366-32144-0/krb5kdc/oryx_127.31.100.1.keytab
--principal=oryx/127.31.100.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.100.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786512017613366-32144-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.100.1
--webserver_port=0
--tserver_master_addrs=127.31.100.62:35003
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9830
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.100.1
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786512017613366-32144-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 04:20:34 UTC on ccycloud.quasar-phqdqp.root.comops.site
build id 4581871
I20260812 05:20:45.131242 9830 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 05:20:45.131405 9830 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 05:20:45.135994 9839 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 05:20:45.136035 9845 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 05:20:45.136001 9836 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 05:20:45.136034 9830 server_base.cc:1034] running on GCE node
I20260812 05:20:45.136356 9830 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 05:20:45.136572 9830 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 05:20:45.137709 9830 hybrid_clock.cc:648] HybridClock initialized: now 1786512045137696 us; error 27 us; skew 500 ppm
Aug 12 05:20:45 dist-test-slave-42z9 krb5kdc[9665](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512045, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.100.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 05:20:45.140004 9830 init.cc:377] Logged in from keytab as oryx/127.31.100.1@KRBTEST.COM (short username oryx)
I20260812 05:20:45.140383 9830 webserver.cc:460] Webserver started at http://127.31.100.1:36363/ using document root <none> and password file <none>
I20260812 05:20:45.140573 9830 fs_manager.cc:359] Metadata directory not provided
I20260812 05:20:45.140619 9830 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 05:20:45.140740 9830 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 05:20:45.141746 9830 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786512017613366-32144-0/minicluster-data/ts-0/data/instance:
uuid: "f54f92d5624f42a9b11d653eb7bd017c"
format_stamp: "Formatted at 2026-08-12 05:20:45 on dist-test-slave-42z9"
server_key: "acd3bb1933c8cbe38c60098ac4cb0ebe"
server_key_iv: "e5c113420c30cb2c4f92ac983df3eb1f"
server_key_version: "encryptionkey@0"
I20260812 05:20:45.142140 9830 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786512017613366-32144-0/minicluster-data/ts-0/wal/instance:
uuid: "f54f92d5624f42a9b11d653eb7bd017c"
format_stamp: "Formatted at 2026-08-12 05:20:45 on dist-test-slave-42z9"
server_key: "acd3bb1933c8cbe38c60098ac4cb0ebe"
server_key_iv: "e5c113420c30cb2c4f92ac983df3eb1f"
server_key_version: "encryptionkey@0"
I20260812 05:20:45.143323 9830 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260812 05:20:45.143962 9857 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 05:20:45.144101 9830 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260812 05:20:45.144184 9830 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786512017613366-32144-0/minicluster-data/ts-0/data,/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786512017613366-32144-0/minicluster-data/ts-0/wal
uuid: "f54f92d5624f42a9b11d653eb7bd017c"
format_stamp: "Formatted at 2026-08-12 05:20:45 on dist-test-slave-42z9"
server_key: "acd3bb1933c8cbe38c60098ac4cb0ebe"
server_key_iv: "e5c113420c30cb2c4f92ac983df3eb1f"
server_key_version: "encryptionkey@0"
I20260812 05:20:45.144239 9830 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786512017613366-32144-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786512017613366-32144-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786512017613366-32144-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 05:20:45.159727 9830 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 05:20:45.162590 9830 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 05:20:45.162730 9830 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 05:20:45.164968 9830 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 05:20:45.165290 9830 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 05:20:45.165324 9830 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 05:20:45.165354 9830 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 05:20:45.165370 9830 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 05:20:45.172629 9830 rpc_server.cc:307] RPC server started. Bound to: 127.31.100.1:44127
I20260812 05:20:45.172693 10031 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.100.1:44127 every 8 connection(s)
I20260812 05:20:45.173054 9830 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786512017613366-32144-0/minicluster-data/ts-0/data/info.pb
I20260812 05:20:45.178023 32144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0waFvT/build/release/bin/kudu as pid 9830
I20260812 05:20:45.178126 32144 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786512017613366-32144-0/minicluster-data/ts-0/wal/instance
I20260812 05:20:45.178340 32144 external_mini_cluster.cc:1428] Setting key 86f9913319e2e1c9a64a23a0eee12494
Aug 12 05:20:45 dist-test-slave-42z9 krb5kdc[9665](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512045, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.100.1@KRBTEST.COM for oryx/127.31.100.62@KRBTEST.COM
WARNING: no policy specified for oryx/127.31.100.2@KRBTEST.COM; defaulting to no policy
I20260812 05:20:45.189657 9802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:45.173763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.100.1:58441 (local address 127.31.100.62:35003)
0812 05:20:45.173812 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0812 05:20:45.173814 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 05:20:45.174107 (+ 293us) server_negotiation.cc:407] Connection header received
0812 05:20:45.174891 (+ 784us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:20:45.174893 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:20:45.174922 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:20:45.174956 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:20:45.176510 (+ 1554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:45.177142 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:45.178374 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:45.178646 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:45.178704 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:45.181158 (+ 2454us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 05:20:45.181160 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 05:20:45.181160 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 05:20:45.181168 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0812 05:20:45.186339 (+ 5171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:20:45.188534 (+ 2195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:20:45.188536 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:20:45.188536 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:20:45.188597 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:20:45.188850 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:20:45.188851 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:20:45.188852 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:20:45.189013 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 05:20:45.189079 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0812 05:20:45.189398 (+ 319us) server_negotiation.cc:299] Negotiation successful
0812 05:20:45.189563 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260812 05:20:45.189774 10038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:45.173757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.62:35003 (local address 127.31.100.1:58441)
0812 05:20:45.173961 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0812 05:20:45.174011 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0812 05:20:45.174750 (+ 739us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:20:45.175184 (+ 434us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:20:45.175190 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:20:45.176092 (+ 902us) client_negotiation.cc:189] Negotiated authn=SASL
0812 05:20:45.176408 (+ 316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:20:45.176415 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:20:45.177229 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:20:45.177230 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:20:45.178293 (+ 1063us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:20:45.178296 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:20:45.178803 (+ 507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:20:45.178804 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:20:45.178947 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:45.179502 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 05:20:45.179512 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0812 05:20:45.181026 (+ 1514us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 05:20:45.186415 (+ 5389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 05:20:45.186423 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 05:20:45.186425 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 05:20:45.186707 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 05:20:45.188656 (+ 1949us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 05:20:45.188657 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 05:20:45.188657 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 05:20:45.188786 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 05:20:45.189084 (+ 298us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 05:20:45.189092 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 05:20:45.189255 (+ 163us) client_negotiation.cc:769] Sending connection context
0812 05:20:45.189369 (+ 114us) client_negotiation.cc:240] Negotiation successful
0812 05:20:45.189553 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":96,"threads_started":1}
I20260812 05:20:45.190286 10034 heartbeater.cc:347] Connected to a master server at 127.31.100.62:35003
I20260812 05:20:45.190392 10034 heartbeater.cc:464] Registering TS with master...
I20260812 05:20:45.190563 10034 heartbeater.cc:511] Master 127.31.100.62:35003 requested a full tablet report, sending...
I20260812 05:20:45.191160 9740 ts_manager.cc:194] Registered new tserver with Master: f54f92d5624f42a9b11d653eb7bd017c (127.31.100.1:44127)
I20260812 05:20:45.191785 9740 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.31.100.1@KRBTEST.COM'} at 127.31.100.1:58441
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.31.100.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.31.100.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786512017613366-32144-0/krb5kdc/oryx_127.31.100.2.keytab.
Entry for principal oryx/127.31.100.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786512017613366-32144-0/krb5kdc/oryx_127.31.100.2.keytab.
I20260812 05:20:45.205761 32144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0waFvT/build/release/bin/kudu
/tmp/dist-test-task0waFvT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786512017613366-32144-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786512017613366-32144-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786512017613366-32144-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786512017613366-32144-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.100.2:0
--local_ip_for_outbound_sockets=127.31.100.2
--webserver_interface=127.31.100.2
--webserver_port=0
--tserver_master_addrs=127.31.100.62:35003
--keytab_file=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786512017613366-32144-0/krb5kdc/oryx_127.31.100.2.keytab
--principal=oryx/127.31.100.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.100.20:44403
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786512017613366-32144-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786512017613366-32144-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786512017613366-32144-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 05:20:45.286077 10046 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 05:20:45.286270 10046 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 05:20:45.286293 10046 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 05:20:45.287354 10046 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 05:20:45.287395 10046 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 05:20:45.287420 10046 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.100.2
I20260812 05:20:45.288800 10046 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.100.20:44403
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786512017613366-32144-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786512017613366-32144-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786512017613366-32144-0/krb5kdc/oryx_127.31.100.2.keytab
--principal=oryx/127.31.100.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.100.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786512017613366-32144-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.100.2
--webserver_port=0
--tserver_master_addrs=127.31.100.62:35003
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10046
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.100.2
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786512017613366-32144-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 04:20:34 UTC on ccycloud.quasar-phqdqp.root.comops.site
build id 4581871
I20260812 05:20:45.288992 10046 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 05:20:45.289161 10046 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 05:20:45.293864 10061 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 05:20:45.293870 10063 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 05:20:45.294028 10046 server_base.cc:1034] running on GCE node
W20260812 05:20:45.293888 10060 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 05:20:45.294322 10046 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 05:20:45.294584 10046 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 05:20:45.295735 10046 hybrid_clock.cc:648] HybridClock initialized: now 1786512045295720 us; error 30 us; skew 500 ppm
Aug 12 05:20:45 dist-test-slave-42z9 krb5kdc[9665](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512045, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.100.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 05:20:45.298126 10046 init.cc:377] Logged in from keytab as oryx/127.31.100.2@KRBTEST.COM (short username oryx)
I20260812 05:20:45.298506 10046 webserver.cc:460] Webserver started at http://127.31.100.2:39859/ using document root <none> and password file <none>
I20260812 05:20:45.298693 10046 fs_manager.cc:359] Metadata directory not provided
I20260812 05:20:45.298732 10046 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 05:20:45.298821 10046 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 05:20:45.299804 10046 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786512017613366-32144-0/minicluster-data/ts-1/data/instance:
uuid: "bf52456a845341ebbf4e4e7405844a2d"
format_stamp: "Formatted at 2026-08-12 05:20:45 on dist-test-slave-42z9"
server_key: "a3f014d8371e850c3a70993559019f9b"
server_key_iv: "8d0f9de869250bcb379c7d82f5d7421e"
server_key_version: "encryptionkey@0"
I20260812 05:20:45.300140 10046 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786512017613366-32144-0/minicluster-data/ts-1/wal/instance:
uuid: "bf52456a845341ebbf4e4e7405844a2d"
format_stamp: "Formatted at 2026-08-12 05:20:45 on dist-test-slave-42z9"
server_key: "a3f014d8371e850c3a70993559019f9b"
server_key_iv: "8d0f9de869250bcb379c7d82f5d7421e"
server_key_version: "encryptionkey@0"
I20260812 05:20:45.301479 10046 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260812 05:20:45.302158 10078 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 05:20:45.302289 10046 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260812 05:20:45.302362 10046 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786512017613366-32144-0/minicluster-data/ts-1/data,/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786512017613366-32144-0/minicluster-data/ts-1/wal
uuid: "bf52456a845341ebbf4e4e7405844a2d"
format_stamp: "Formatted at 2026-08-12 05:20:45 on dist-test-slave-42z9"
server_key: "a3f014d8371e850c3a70993559019f9b"
server_key_iv: "8d0f9de869250bcb379c7d82f5d7421e"
server_key_version: "encryptionkey@0"
I20260812 05:20:45.302438 10046 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786512017613366-32144-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786512017613366-32144-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786512017613366-32144-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 05:20:45.335669 10046 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 05:20:45.337862 10046 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 05:20:45.337980 10046 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 05:20:45.338183 10046 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 05:20:45.338444 10046 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 05:20:45.338474 10046 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 05:20:45.338495 10046 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 05:20:45.338510 10046 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 05:20:45.344015 10046 rpc_server.cc:307] RPC server started. Bound to: 127.31.100.2:36081
I20260812 05:20:45.344097 10255 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.100.2:36081 every 8 connection(s)
I20260812 05:20:45.344355 10046 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786512017613366-32144-0/minicluster-data/ts-1/data/info.pb
Aug 12 05:20:45 dist-test-slave-42z9 krb5kdc[9665](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512045, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.100.2@KRBTEST.COM for oryx/127.31.100.62@KRBTEST.COM
I20260812 05:20:45.350456 32144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0waFvT/build/release/bin/kudu as pid 10046
I20260812 05:20:45.350569 32144 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786512017613366-32144-0/minicluster-data/ts-1/wal/instance
I20260812 05:20:45.350764 32144 external_mini_cluster.cc:1428] Setting key 89da3ef21d34af26105ab31f732bb5b1
I20260812 05:20:45.353391 9802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:45.345013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.100.2:46809 (local address 127.31.100.62:35003)
0812 05:20:45.345058 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0812 05:20:45.345060 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 05:20:45.345254 (+ 194us) server_negotiation.cc:407] Connection header received
0812 05:20:45.345748 (+ 494us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:20:45.345749 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:20:45.345762 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:20:45.345812 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:20:45.346725 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:45.347227 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:45.348290 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:45.348480 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:45.348532 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:45.350652 (+ 2120us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 05:20:45.350653 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 05:20:45.350653 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 05:20:45.350662 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0812 05:20:45.352097 (+ 1435us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:20:45.352534 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:20:45.352535 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:20:45.352535 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:20:45.352603 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:20:45.352813 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:20:45.352814 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:20:45.352814 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:20:45.352939 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 05:20:45.352997 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0812 05:20:45.353213 (+ 216us) server_negotiation.cc:299] Negotiation successful
0812 05:20:45.353300 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260812 05:20:45.353595 10258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:45.344990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.62:35003 (local address 127.31.100.2:46809)
0812 05:20:45.345127 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0812 05:20:45.345161 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0812 05:20:45.345669 (+ 508us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:20:45.345865 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:20:45.345868 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:20:45.346398 (+ 530us) client_negotiation.cc:189] Negotiated authn=SASL
0812 05:20:45.346626 (+ 228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:20:45.346632 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:20:45.347308 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:20:45.347308 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:20:45.348218 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:20:45.348221 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:20:45.348543 (+ 322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:20:45.348544 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:20:45.348653 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:45.349108 (+ 455us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 05:20:45.349115 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0812 05:20:45.350552 (+ 1437us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 05:20:45.352175 (+ 1623us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 05:20:45.352181 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 05:20:45.352183 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 05:20:45.352459 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 05:20:45.352656 (+ 197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 05:20:45.352657 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 05:20:45.352657 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 05:20:45.352763 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 05:20:45.352995 (+ 232us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 05:20:45.352998 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 05:20:45.353107 (+ 109us) client_negotiation.cc:769] Sending connection context
0812 05:20:45.353180 (+ 73us) client_negotiation.cc:240] Negotiation successful
0812 05:20:45.353341 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":77,"threads_started":1}
I20260812 05:20:45.354103 10256 heartbeater.cc:347] Connected to a master server at 127.31.100.62:35003
I20260812 05:20:45.354202 10256 heartbeater.cc:464] Registering TS with master...
I20260812 05:20:45.354384 10256 heartbeater.cc:511] Master 127.31.100.62:35003 requested a full tablet report, sending...
I20260812 05:20:45.354732 9740 ts_manager.cc:194] Registered new tserver with Master: bf52456a845341ebbf4e4e7405844a2d (127.31.100.2:36081)
I20260812 05:20:45.355080 9740 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.31.100.2@KRBTEST.COM'} at 127.31.100.2:46809
WARNING: no policy specified for oryx/127.31.100.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.31.100.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.31.100.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786512017613366-32144-0/krb5kdc/oryx_127.31.100.3.keytab.
Entry for principal oryx/127.31.100.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786512017613366-32144-0/krb5kdc/oryx_127.31.100.3.keytab.
I20260812 05:20:45.375679 32144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0waFvT/build/release/bin/kudu
/tmp/dist-test-task0waFvT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786512017613366-32144-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786512017613366-32144-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786512017613366-32144-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786512017613366-32144-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.100.3:0
--local_ip_for_outbound_sockets=127.31.100.3
--webserver_interface=127.31.100.3
--webserver_port=0
--tserver_master_addrs=127.31.100.62:35003
--keytab_file=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786512017613366-32144-0/krb5kdc/oryx_127.31.100.3.keytab
--principal=oryx/127.31.100.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.100.20:44403
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786512017613366-32144-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786512017613366-32144-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786512017613366-32144-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 05:20:45.455508 10264 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 05:20:45.455700 10264 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 05:20:45.455724 10264 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 05:20:45.456784 10264 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 05:20:45.456818 10264 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 05:20:45.456843 10264 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.100.3
I20260812 05:20:45.458297 10264 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.100.20:44403
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786512017613366-32144-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786512017613366-32144-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786512017613366-32144-0/krb5kdc/oryx_127.31.100.3.keytab
--principal=oryx/127.31.100.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.100.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786512017613366-32144-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.100.3
--webserver_port=0
--tserver_master_addrs=127.31.100.62:35003
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10264
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.100.3
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786512017613366-32144-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 04:20:34 UTC on ccycloud.quasar-phqdqp.root.comops.site
build id 4581871
I20260812 05:20:45.458483 10264 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 05:20:45.458665 10264 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 05:20:45.463406 10273 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 05:20:45.463461 10280 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 05:20:45.463483 10274 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 05:20:45.463605 10264 server_base.cc:1034] running on GCE node
I20260812 05:20:45.463827 10264 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 05:20:45.464021 10264 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 05:20:45.465165 10264 hybrid_clock.cc:648] HybridClock initialized: now 1786512045465154 us; error 28 us; skew 500 ppm
Aug 12 05:20:45 dist-test-slave-42z9 krb5kdc[9665](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512045, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.100.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 05:20:45.467736 10264 init.cc:377] Logged in from keytab as oryx/127.31.100.3@KRBTEST.COM (short username oryx)
I20260812 05:20:45.468153 10264 webserver.cc:460] Webserver started at http://127.31.100.3:39337/ using document root <none> and password file <none>
I20260812 05:20:45.468369 10264 fs_manager.cc:359] Metadata directory not provided
I20260812 05:20:45.468411 10264 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 05:20:45.468537 10264 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 05:20:45.469568 10264 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786512017613366-32144-0/minicluster-data/ts-2/data/instance:
uuid: "167b45749e0d4747866e5be16306875f"
format_stamp: "Formatted at 2026-08-12 05:20:45 on dist-test-slave-42z9"
server_key: "f7161c65bf753427bc7aa1056c49ec23"
server_key_iv: "e519440e368b175644e8c7a995ac3470"
server_key_version: "encryptionkey@0"
I20260812 05:20:45.469938 10264 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786512017613366-32144-0/minicluster-data/ts-2/wal/instance:
uuid: "167b45749e0d4747866e5be16306875f"
format_stamp: "Formatted at 2026-08-12 05:20:45 on dist-test-slave-42z9"
server_key: "f7161c65bf753427bc7aa1056c49ec23"
server_key_iv: "e519440e368b175644e8c7a995ac3470"
server_key_version: "encryptionkey@0"
I20260812 05:20:45.471158 10264 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260812 05:20:45.471855 10289 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 05:20:45.472015 10264 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260812 05:20:45.472097 10264 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786512017613366-32144-0/minicluster-data/ts-2/data,/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786512017613366-32144-0/minicluster-data/ts-2/wal
uuid: "167b45749e0d4747866e5be16306875f"
format_stamp: "Formatted at 2026-08-12 05:20:45 on dist-test-slave-42z9"
server_key: "f7161c65bf753427bc7aa1056c49ec23"
server_key_iv: "e519440e368b175644e8c7a995ac3470"
server_key_version: "encryptionkey@0"
I20260812 05:20:45.472162 10264 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786512017613366-32144-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786512017613366-32144-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786512017613366-32144-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 05:20:45.489602 10264 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 05:20:45.491731 10264 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 05:20:45.491873 10264 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 05:20:45.492102 10264 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 05:20:45.492412 10264 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 05:20:45.492447 10264 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 05:20:45.492477 10264 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 05:20:45.492499 10264 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 05:20:45.498080 10264 rpc_server.cc:307] RPC server started. Bound to: 127.31.100.3:39085
I20260812 05:20:45.498119 10465 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.100.3:39085 every 8 connection(s)
I20260812 05:20:45.498430 10264 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786512017613366-32144-0/minicluster-data/ts-2/data/info.pb
I20260812 05:20:45.500264 32144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0waFvT/build/release/bin/kudu as pid 10264
I20260812 05:20:45.500350 32144 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786512017613366-32144-0/minicluster-data/ts-2/wal/instance
I20260812 05:20:45.500540 32144 external_mini_cluster.cc:1428] Setting key dd3c364f955f1e0d96508b2f4663c609
Aug 12 05:20:45 dist-test-slave-42z9 krb5kdc[9665](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512045, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.100.3@KRBTEST.COM for oryx/127.31.100.62@KRBTEST.COM
I20260812 05:20:45.507225 9802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:45.499109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.100.3:45423 (local address 127.31.100.62:35003)
0812 05:20:45.499157 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0812 05:20:45.499160 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 05:20:45.499356 (+ 196us) server_negotiation.cc:407] Connection header received
0812 05:20:45.499903 (+ 547us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:20:45.499904 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:20:45.499915 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:20:45.499971 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:20:45.501173 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:45.501821 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:45.502768 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:45.502958 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:45.503012 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:45.504965 (+ 1953us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 05:20:45.504966 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 05:20:45.504967 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 05:20:45.504973 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0812 05:20:45.506145 (+ 1172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:20:45.506525 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:20:45.506526 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:20:45.506526 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:20:45.506579 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:20:45.506763 (+ 184us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:20:45.506764 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:20:45.506764 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:20:45.506853 (+ 89us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 05:20:45.506898 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0812 05:20:45.507107 (+ 209us) server_negotiation.cc:299] Negotiation successful
0812 05:20:45.507164 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260812 05:20:45.507400 10468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:45.499085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.62:35003 (local address 127.31.100.3:45423)
0812 05:20:45.499245 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0812 05:20:45.499280 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0812 05:20:45.499791 (+ 511us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:20:45.500028 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:20:45.500031 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:20:45.500807 (+ 776us) client_negotiation.cc:189] Negotiated authn=SASL
0812 05:20:45.501092 (+ 285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:20:45.501100 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:20:45.501900 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:20:45.501901 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:20:45.502694 (+ 793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:20:45.502696 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:20:45.503019 (+ 323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:20:45.503019 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:20:45.503112 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:45.503570 (+ 458us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 05:20:45.503578 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0812 05:20:45.504876 (+ 1298us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 05:20:45.506227 (+ 1351us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 05:20:45.506236 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 05:20:45.506238 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 05:20:45.506450 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 05:20:45.506624 (+ 174us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 05:20:45.506624 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 05:20:45.506624 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 05:20:45.506721 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 05:20:45.506894 (+ 173us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 05:20:45.506896 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 05:20:45.507014 (+ 118us) client_negotiation.cc:769] Sending connection context
0812 05:20:45.507089 (+ 75us) client_negotiation.cc:240] Negotiation successful
0812 05:20:45.507202 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":88,"threads_started":1}
I20260812 05:20:45.507815 10466 heartbeater.cc:347] Connected to a master server at 127.31.100.62:35003
I20260812 05:20:45.507897 10466 heartbeater.cc:464] Registering TS with master...
I20260812 05:20:45.508044 10466 heartbeater.cc:511] Master 127.31.100.62:35003 requested a full tablet report, sending...
I20260812 05:20:45.508361 9740 ts_manager.cc:194] Registered new tserver with Master: 167b45749e0d4747866e5be16306875f (127.31.100.3:39085)
I20260812 05:20:45.508708 9740 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.31.100.3@KRBTEST.COM'} at 127.31.100.3:45423
I20260812 05:20:45.512176 32144 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.31.100.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.31.100.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.31.100.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786512017613366-32144-0/krb5kdc/oryx_127.31.100.61.keytab.
Entry for principal oryx/127.31.100.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786512017613366-32144-0/krb5kdc/oryx_127.31.100.61.keytab.
W20260812 05:20:45.622702 32144 mini_kdc.cc:262] Time spent creating service keytab for oryx/127.31.100.61: real 0.110s user 0.000s sys 0.003s
I20260812 05:20:45.623083 32144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0waFvT/build/release/bin/kudu
/tmp/dist-test-task0waFvT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786512017613366-32144-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786512017613366-32144-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786512017613366-32144-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786512017613366-32144-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.100.61:34209
--webserver_interface=127.31.100.61
--webserver_port=0
--keytab_file=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786512017613366-32144-0/krb5kdc/oryx_127.31.100.61.keytab
--principal=oryx/127.31.100.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.100.20:44403
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.100.62:35003,127.31.100.61:34209
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786512017613366-32144-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786512017613366-32144-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786512017613366-32144-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 05:20:45.708796 10473 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 05:20:45.709004 10473 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 05:20:45.709028 10473 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 05:20:45.710199 10473 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 05:20:45.710240 10473 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 05:20:45.710253 10473 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 05:20:45.710266 10473 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 05:20:45.710289 10473 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 05:20:45.711982 10473 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.100.20:44403
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786512017613366-32144-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786512017613366-32144-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.31.100.62:35003,127.31.100.61:34209
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786512017613366-32144-0/krb5kdc/oryx_127.31.100.61.keytab
--principal=oryx/127.31.100.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.100.61:34209
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786512017613366-32144-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.31.100.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10473
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786512017613366-32144-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 04:20:34 UTC on ccycloud.quasar-phqdqp.root.comops.site
build id 4581871
W20260812 05:20:45.712090 10473 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.31.100.62:35003,127.31.100.61:34209'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260812 05:20:45.712235 10473 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 05:20:45.712433 10473 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 05:20:45.716971 10485 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 05:20:45.717036 10483 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 05:20:45.717061 10481 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 05:20:45.717131 10473 server_base.cc:1034] running on GCE node
I20260812 05:20:45.717370 10473 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 05:20:45.717679 10473 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 05:20:45.718811 10473 hybrid_clock.cc:648] HybridClock initialized: now 1786512045718793 us; error 34 us; skew 500 ppm
Aug 12 05:20:45 dist-test-slave-42z9 krb5kdc[9665](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512045, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.100.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 05:20:45.721225 10473 init.cc:377] Logged in from keytab as oryx/127.31.100.61@KRBTEST.COM (short username oryx)
I20260812 05:20:45.721644 10473 webserver.cc:460] Webserver started at http://127.31.100.61:33485/ using document root <none> and password file <none>
I20260812 05:20:45.721863 10473 fs_manager.cc:359] Metadata directory not provided
I20260812 05:20:45.721907 10473 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 05:20:45.722010 10473 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 05:20:45.722926 10473 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786512017613366-32144-0/minicluster-data/master-1/data/instance:
uuid: "e8be870d7089424d82f5be0171ba5242"
format_stamp: "Formatted at 2026-08-12 05:20:45 on dist-test-slave-42z9"
server_key: "f837a7bd167f164bae5ce9ac1c4f2604"
server_key_iv: "c82ec8da91b1eed7a44e69b1c4c4d483"
server_key_version: "encryptionkey@0"
I20260812 05:20:45.723244 10473 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786512017613366-32144-0/minicluster-data/master-1/wal/instance:
uuid: "e8be870d7089424d82f5be0171ba5242"
format_stamp: "Formatted at 2026-08-12 05:20:45 on dist-test-slave-42z9"
server_key: "f837a7bd167f164bae5ce9ac1c4f2604"
server_key_iv: "c82ec8da91b1eed7a44e69b1c4c4d483"
server_key_version: "encryptionkey@0"
I20260812 05:20:45.724445 10473 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260812 05:20:45.725132 10501 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 05:20:45.725279 10473 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260812 05:20:45.725353 10473 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786512017613366-32144-0/minicluster-data/master-1/data,/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786512017613366-32144-0/minicluster-data/master-1/wal
uuid: "e8be870d7089424d82f5be0171ba5242"
format_stamp: "Formatted at 2026-08-12 05:20:45 on dist-test-slave-42z9"
server_key: "f837a7bd167f164bae5ce9ac1c4f2604"
server_key_iv: "c82ec8da91b1eed7a44e69b1c4c4d483"
server_key_version: "encryptionkey@0"
I20260812 05:20:45.725414 10473 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786512017613366-32144-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786512017613366-32144-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786512017613366-32144-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 05:20:45.732841 10473 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 05:20:45.734754 10473 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 05:20:45.734894 10473 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 05:20:45.738963 10473 rpc_server.cc:307] RPC server started. Bound to: 127.31.100.61:34209
I20260812 05:20:45.739004 10584 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.100.61:34209 every 8 connection(s)
I20260812 05:20:45.739300 10473 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786512017613366-32144-0/minicluster-data/master-1/data/info.pb
I20260812 05:20:45.740259 10585 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 05:20:45.741748 10585 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.100.62" port: 35003 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 05:20:45 dist-test-slave-42z9 krb5kdc[9665](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512045, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.100.61@KRBTEST.COM for oryx/127.31.100.62@KRBTEST.COM
I20260812 05:20:45.749022 32144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0waFvT/build/release/bin/kudu as pid 10473
I20260812 05:20:45.750190 9802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:45.742477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48176 (local address 127.31.100.62:35003)
0812 05:20:45.742539 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0812 05:20:45.742541 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 05:20:45.742717 (+ 176us) server_negotiation.cc:407] Connection header received
0812 05:20:45.743158 (+ 441us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:20:45.743159 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:20:45.743169 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:20:45.743222 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:20:45.743796 (+ 574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:45.744306 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:45.745240 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:45.745475 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:45.745530 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:45.747490 (+ 1960us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 05:20:45.747491 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 05:20:45.747491 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 05:20:45.747498 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 05:20:45.748710 (+ 1212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:20:45.749151 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:20:45.749152 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:20:45.749153 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:20:45.749209 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:20:45.749484 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:20:45.749485 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:20:45.749486 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:20:45.749689 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 05:20:45.749748 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0812 05:20:45.750065 (+ 317us) server_negotiation.cc:299] Negotiation successful
0812 05:20:45.750128 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260812 05:20:45.750422 10587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:45.742388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.62:35003 (local address 127.0.0.1:48176)
0812 05:20:45.742607 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0812 05:20:45.742640 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0812 05:20:45.743080 (+ 440us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:20:45.743280 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:20:45.743283 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:20:45.743527 (+ 244us) client_negotiation.cc:189] Negotiated authn=SASL
0812 05:20:45.743712 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:20:45.743717 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:20:45.744378 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:20:45.744379 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:20:45.745163 (+ 784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:20:45.745165 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:20:45.745539 (+ 374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:20:45.745540 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:20:45.745636 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:45.746047 (+ 411us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 05:20:45.746054 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0812 05:20:45.747391 (+ 1337us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 05:20:45.748777 (+ 1386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 05:20:45.748783 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 05:20:45.748785 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 05:20:45.749069 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 05:20:45.749273 (+ 204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 05:20:45.749273 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 05:20:45.749274 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 05:20:45.749381 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 05:20:45.749755 (+ 374us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 05:20:45.749759 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 05:20:45.749925 (+ 166us) client_negotiation.cc:769] Sending connection context
0812 05:20:45.750040 (+ 115us) client_negotiation.cc:240] Negotiation successful
0812 05:20:45.750200 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":100,"threads_started":1}
I20260812 05:20:45.751091 10585 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.100.61" port: 34209 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 05:20:45 dist-test-slave-42z9 krb5kdc[9665](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512044, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.31.100.61@KRBTEST.COM
Aug 12 05:20:45 dist-test-slave-42z9 krb5kdc[9665](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512045, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.100.61@KRBTEST.COM for oryx/127.31.100.61@KRBTEST.COM
I20260812 05:20:45.758149 10587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:45.751338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.61:34209 (local address 127.0.0.1:49834)
0812 05:20:45.751522 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0812 05:20:45.751530 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0812 05:20:45.751572 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:20:45.751678 (+ 106us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:20:45.751678 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:20:45.751805 (+ 127us) client_negotiation.cc:189] Negotiated authn=SASL
0812 05:20:45.751867 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:20:45.751868 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:20:45.752433 (+ 565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:20:45.752435 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:20:45.752971 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:20:45.752974 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:20:45.753407 (+ 433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:20:45.753407 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:20:45.753501 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:45.753808 (+ 307us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 05:20:45.753813 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0812 05:20:45.755285 (+ 1472us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 05:20:45.757013 (+ 1728us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 05:20:45.757014 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 05:20:45.757014 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 05:20:45.757279 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 05:20:45.757502 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 05:20:45.757503 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 05:20:45.757503 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 05:20:45.757553 (+ 50us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 05:20:45.757851 (+ 298us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 05:20:45.757852 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 05:20:45.757888 (+ 36us) client_negotiation.cc:769] Sending connection context
0812 05:20:45.757938 (+ 50us) client_negotiation.cc:240] Negotiation successful
0812 05:20:45.758036 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":173}
I20260812 05:20:45.758237 10593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:45.751402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49834 (local address 127.31.100.61:34209)
0812 05:20:45.751553 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0812 05:20:45.751554 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0812 05:20:45.751562 (+ 8us) server_negotiation.cc:407] Connection header received
0812 05:20:45.751620 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:20:45.751620 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:20:45.751629 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:20:45.751673 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:20:45.751925 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:45.752365 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:45.753053 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:45.753337 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:45.753413 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:45.755361 (+ 1948us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 05:20:45.755362 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 05:20:45.755362 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 05:20:45.755368 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0812 05:20:45.756932 (+ 1564us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:20:45.757346 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:20:45.757349 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:20:45.757349 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:20:45.757416 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:20:45.757605 (+ 189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:20:45.757605 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:20:45.757606 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:20:45.757772 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 05:20:45.757868 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0812 05:20:45.757983 (+ 115us) server_negotiation.cc:299] Negotiation successful
0812 05:20:45.758098 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":37,"server-negotiator.queue_time_us":103,"thread_start_us":66,"threads_started":1}
I20260812 05:20:45.758347 10590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:45.749528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49832 (local address 127.31.100.61:34209)
0812 05:20:45.749669 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0812 05:20:45.749674 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 05:20:45.749814 (+ 140us) server_negotiation.cc:407] Connection header received
0812 05:20:45.749839 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:20:45.749841 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:20:45.749852 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:20:45.749921 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:20:45.750611 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:45.751512 (+ 901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:45.752531 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:45.752810 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:45.752860 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:45.755043 (+ 2183us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 05:20:45.755052 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 05:20:45.755052 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 05:20:45.755059 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 05:20:45.757320 (+ 2261us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:20:45.757631 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:20:45.757632 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:20:45.757632 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:20:45.757673 (+ 41us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:20:45.757828 (+ 155us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:20:45.757829 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:20:45.757829 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:20:45.757955 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 05:20:45.758088 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0812 05:20:45.758185 (+ 97us) server_negotiation.cc:299] Negotiation successful
0812 05:20:45.758252 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111,"thread_start_us":76,"threads_started":1}
I20260812 05:20:45.759673 10585 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e8be870d7089424d82f5be0171ba5242: Bootstrap starting.
I20260812 05:20:45.760277 10585 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e8be870d7089424d82f5be0171ba5242: Neither blocks nor log segments found. Creating new log.
I20260812 05:20:45.760528 10585 log.cc:824] T 00000000000000000000000000000000 P e8be870d7089424d82f5be0171ba5242: Log is configured to *not* fsync() on all Append() calls
I20260812 05:20:45.761147 10585 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e8be870d7089424d82f5be0171ba5242: No bootstrap required, opened a new log
I20260812 05:20:45.762538 10585 raft_consensus.cc:348] T 00000000000000000000000000000000 P e8be870d7089424d82f5be0171ba5242 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "070ed78c702d4333919d5d31da7b7608" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 35003 } } peers { permanent_uuid: "e8be870d7089424d82f5be0171ba5242" member_type: VOTER last_known_addr { host: "127.31.100.61" port: 34209 } }
I20260812 05:20:45.762652 10585 raft_consensus.cc:374] T 00000000000000000000000000000000 P e8be870d7089424d82f5be0171ba5242 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 05:20:45.762672 10585 raft_consensus.cc:729] T 00000000000000000000000000000000 P e8be870d7089424d82f5be0171ba5242 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8be870d7089424d82f5be0171ba5242, State: Initialized, Role: FOLLOWER
I20260812 05:20:45.762763 10585 consensus_queue.cc:260] T 00000000000000000000000000000000 P e8be870d7089424d82f5be0171ba5242 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "070ed78c702d4333919d5d31da7b7608" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 35003 } } peers { permanent_uuid: "e8be870d7089424d82f5be0171ba5242" member_type: VOTER last_known_addr { host: "127.31.100.61" port: 34209 } }
I20260812 05:20:45.763016 10585 sys_catalog.cc:564] T 00000000000000000000000000000000 P e8be870d7089424d82f5be0171ba5242 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 05:20:45.762985 10597 sys_catalog.cc:455] T 00000000000000000000000000000000 P e8be870d7089424d82f5be0171ba5242 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "070ed78c702d4333919d5d31da7b7608" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 35003 } } peers { permanent_uuid: "e8be870d7089424d82f5be0171ba5242" member_type: VOTER last_known_addr { host: "127.31.100.61" port: 34209 } } }
I20260812 05:20:45.763073 10597 sys_catalog.cc:458] T 00000000000000000000000000000000 P e8be870d7089424d82f5be0171ba5242 [sys.catalog]: This master's current role is: FOLLOWER
W20260812 05:20:45.764707 10612 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e8be870d7089424d82f5be0171ba5242: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 05:20:45.764791 10612 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 05:20:45.766086 10473 master_runner.cc:428] Detected that this master e8be870d7089424d82f5be0171ba5242 is joining an existing cluster
I20260812 05:20:45.766139 10473 master_runner.cc:432] Initiating AddMaster RPC to add 127.31.100.61:34209
I20260812 05:20:45.773949 9802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:45.767157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48182 (local address 127.31.100.62:35003)
0812 05:20:45.767210 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0812 05:20:45.767212 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 05:20:45.767345 (+ 133us) server_negotiation.cc:407] Connection header received
0812 05:20:45.767460 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:20:45.767461 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:20:45.767472 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:20:45.767501 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:20:45.768068 (+ 567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:45.768598 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:45.769535 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:45.770011 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:45.770064 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:45.771160 (+ 1096us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 05:20:45.771161 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 05:20:45.771162 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 05:20:45.771168 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0812 05:20:45.772720 (+ 1552us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:20:45.773183 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:20:45.773185 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:20:45.773185 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:20:45.773239 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:20:45.773416 (+ 177us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:20:45.773417 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:20:45.773417 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:20:45.773566 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 05:20:45.773630 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0812 05:20:45.773773 (+ 143us) server_negotiation.cc:299] Negotiation successful
0812 05:20:45.773848 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260812 05:20:45.774180 10624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:45.767331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.61:34209 (local address 127.0.0.1:49844)
0812 05:20:45.767455 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0812 05:20:45.767462 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0812 05:20:45.767512 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:20:45.767653 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:20:45.767653 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:20:45.767895 (+ 242us) client_negotiation.cc:189] Negotiated authn=SASL
0812 05:20:45.767972 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:20:45.767973 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:20:45.769121 (+ 1148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:20:45.769122 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:20:45.770030 (+ 908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:20:45.770033 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:20:45.770529 (+ 496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:20:45.770530 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:20:45.770606 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:45.771192 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 05:20:45.771196 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0812 05:20:45.771486 (+ 290us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 05:20:45.773169 (+ 1683us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 05:20:45.773170 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 05:20:45.773170 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 05:20:45.773386 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 05:20:45.773583 (+ 197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 05:20:45.773584 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 05:20:45.773584 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 05:20:45.773648 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 05:20:45.773871 (+ 223us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 05:20:45.773871 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 05:20:45.773919 (+ 48us) client_negotiation.cc:769] Sending connection context
0812 05:20:45.773987 (+ 68us) client_negotiation.cc:240] Negotiation successful
0812 05:20:45.774080 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"mutex_wait_us":2,"thread_start_us":64,"threads_started":1}
I20260812 05:20:45.774412 10593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:45.767379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49844 (local address 127.31.100.61:34209)
0812 05:20:45.767412 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0812 05:20:45.767414 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 05:20:45.767514 (+ 100us) server_negotiation.cc:407] Connection header received
0812 05:20:45.767572 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:20:45.767573 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:20:45.767582 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:20:45.767643 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:20:45.768315 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:45.769040 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:45.770111 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:45.770437 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:45.770506 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:45.771571 (+ 1065us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 05:20:45.771573 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 05:20:45.771573 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 05:20:45.771579 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0812 05:20:45.773097 (+ 1518us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:20:45.773481 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:20:45.773482 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:20:45.773483 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:20:45.773526 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:20:45.773700 (+ 174us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:20:45.773701 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:20:45.773701 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:20:45.773809 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 05:20:45.773868 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0812 05:20:45.774018 (+ 150us) server_negotiation.cc:299] Negotiation successful
0812 05:20:45.774216 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260812 05:20:45.774574 10623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:45.767086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.62:35003 (local address 127.0.0.1:48182)
0812 05:20:45.767287 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0812 05:20:45.767297 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0812 05:20:45.767384 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:20:45.767582 (+ 198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:20:45.767583 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:20:45.767893 (+ 310us) client_negotiation.cc:189] Negotiated authn=SASL
0812 05:20:45.767972 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:20:45.767973 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:20:45.768671 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:20:45.768671 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:20:45.769313 (+ 642us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:20:45.769315 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:20:45.770160 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:20:45.770160 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:20:45.770206 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:45.770744 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 05:20:45.770748 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0812 05:20:45.771082 (+ 334us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 05:20:45.772808 (+ 1726us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 05:20:45.772809 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 05:20:45.772809 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 05:20:45.773099 (+ 290us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 05:20:45.773293 (+ 194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 05:20:45.773294 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 05:20:45.773294 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 05:20:45.773361 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 05:20:45.773620 (+ 259us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 05:20:45.773621 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 05:20:45.773661 (+ 40us) client_negotiation.cc:769] Sending connection context
0812 05:20:45.774142 (+ 481us) client_negotiation.cc:240] Negotiation successful
0812 05:20:45.774365 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171,"mutex_wait_us":41,"thread_start_us":90,"threads_started":1}
Aug 12 05:20:45 dist-test-slave-42z9 krb5kdc[9665](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512044, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.100.62@KRBTEST.COM for oryx/127.31.100.61@KRBTEST.COM
I20260812 05:20:45.783586 10593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:45.776870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49852 (local address 127.31.100.61:34209)
0812 05:20:45.776959 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0812 05:20:45.776961 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 05:20:45.777040 (+ 79us) server_negotiation.cc:407] Connection header received
0812 05:20:45.777130 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:20:45.777131 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:20:45.777138 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:20:45.777178 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:20:45.777660 (+ 482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:45.778086 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:45.779173 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:45.779338 (+ 165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:45.779412 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:45.781393 (+ 1981us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 05:20:45.781394 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 05:20:45.781394 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 05:20:45.781399 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0812 05:20:45.782554 (+ 1155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:20:45.782961 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:20:45.782961 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:20:45.782961 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:20:45.783001 (+ 40us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:20:45.783182 (+ 181us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:20:45.783183 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:20:45.783183 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:20:45.783261 (+ 78us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 05:20:45.783311 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0812 05:20:45.783481 (+ 170us) server_negotiation.cc:299] Negotiation successful
0812 05:20:45.783533 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260812 05:20:45.783692 10627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:45.776813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.61:34209 (local address 127.0.0.1:49852)
0812 05:20:45.776944 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0812 05:20:45.776991 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0812 05:20:45.777064 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:20:45.777186 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:20:45.777188 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:20:45.777538 (+ 350us) client_negotiation.cc:189] Negotiated authn=SASL
0812 05:20:45.777598 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:20:45.777601 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:20:45.778153 (+ 552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:20:45.778154 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:20:45.779092 (+ 938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:20:45.779095 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:20:45.779406 (+ 311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:20:45.779407 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:20:45.779505 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:45.779878 (+ 373us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 05:20:45.779882 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0812 05:20:45.781310 (+ 1428us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 05:20:45.782628 (+ 1318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 05:20:45.782635 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 05:20:45.782637 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 05:20:45.782886 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 05:20:45.783052 (+ 166us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 05:20:45.783052 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 05:20:45.783053 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 05:20:45.783114 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 05:20:45.783315 (+ 201us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 05:20:45.783326 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 05:20:45.783377 (+ 51us) client_negotiation.cc:769] Sending connection context
0812 05:20:45.783463 (+ 86us) client_negotiation.cc:240] Negotiation successful
0812 05:20:45.783580 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":67,"threads_started":1}
I20260812 05:20:45.784511 9740 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.31.100.61:34209: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "e8be870d7089424d82f5be0171ba5242"
member_type: NON_VOTER
last_known_addr {
host: "127.31.100.61"
port: 34209
}
attrs {
promote: true
}
}
dest_uuid: "070ed78c702d4333919d5d31da7b7608"
cas_config_opid_index: -1
I20260812 05:20:45.784793 9740 consensus_queue.cc:237] T 00000000000000000000000000000000 P 070ed78c702d4333919d5d31da7b7608 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "070ed78c702d4333919d5d31da7b7608" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 35003 } } peers { permanent_uuid: "e8be870d7089424d82f5be0171ba5242" member_type: NON_VOTER last_known_addr { host: "127.31.100.61" port: 34209 } attrs { promote: true } }
I20260812 05:20:45.785218 10629 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 070ed78c702d4333919d5d31da7b7608 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER e8be870d7089424d82f5be0171ba5242 (127.31.100.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "070ed78c702d4333919d5d31da7b7608" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 35003 } } peers { permanent_uuid: "e8be870d7089424d82f5be0171ba5242" member_type: NON_VOTER last_known_addr { host: "127.31.100.61" port: 34209 } attrs { promote: true } } }
I20260812 05:20:45.785524 10543 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e8be870d7089424d82f5be0171ba5242 [term 0 FOLLOWER]: Advancing to term 1
I20260812 05:20:45.827337 10543 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e8be870d7089424d82f5be0171ba5242 [term 1 FOLLOWER]: Refusing update from remote peer 070ed78c702d4333919d5d31da7b7608: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260812 05:20:45.827399 10629 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.31.100.61:34209
I20260812 05:20:45.827649 10632 sys_catalog.cc:455] T 00000000000000000000000000000000 P 070ed78c702d4333919d5d31da7b7608 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "070ed78c702d4333919d5d31da7b7608" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "070ed78c702d4333919d5d31da7b7608" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 35003 } } peers { permanent_uuid: "e8be870d7089424d82f5be0171ba5242" member_type: NON_VOTER last_known_addr { host: "127.31.100.61" port: 34209 } attrs { promote: true } } }
I20260812 05:20:45.827737 10632 sys_catalog.cc:458] T 00000000000000000000000000000000 P 070ed78c702d4333919d5d31da7b7608 [sys.catalog]: This master's current role is: LEADER
I20260812 05:20:45.828104 10473 master.cc:537] Master@127.31.100.61:34209 shutting down...
I20260812 05:20:45.828248 10632 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 070ed78c702d4333919d5d31da7b7608 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8be870d7089424d82f5be0171ba5242" member_type: NON_VOTER last_known_addr { host: "127.31.100.61" port: 34209 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260812 05:20:45.830437 9718 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260812 05:20:45.831996 9718 consensus_peers.cc:487] T 00000000000000000000000000000000 P 070ed78c702d4333919d5d31da7b7608 -> Peer e8be870d7089424d82f5be0171ba5242 (127.31.100.61:34209): Couldn't send request to peer e8be870d7089424d82f5be0171ba5242. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260812 05:20:45.833652 10473 raft_consensus.cc:2229] T 00000000000000000000000000000000 P e8be870d7089424d82f5be0171ba5242 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 05:20:45.833755 10473 raft_consensus.cc:2258] T 00000000000000000000000000000000 P e8be870d7089424d82f5be0171ba5242 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 05:20:45.833808 10473 tablet_replica.cc:330] T 00000000000000000000000000000000 P e8be870d7089424d82f5be0171ba5242: stopping tablet replica
WARNING: no policy specified for oryx/127.31.100.60@KRBTEST.COM; defaulting to no policy
I20260812 05:20:45.846655 10473 master.cc:552] Master@127.31.100.61:34209 shutdown complete.
I20260812 05:20:45.846700 10473 master_runner.cc:305] Clearing existing system tablet
I20260812 05:20:45.846825 10473 fs_manager.cc:359] Metadata directory not provided
I20260812 05:20:45.846876 10473 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 05:20:45.847427 10473 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.000s sys 0.001s
I20260812 05:20:45.847905 10642 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 05:20:45.848022 10473 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786512017613366-32144-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 05:20:45.848074 10473 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260812 05:20:45.848095 10473 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786512017613366-32144-0/minicluster-data/master-1/data,/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786512017613366-32144-0/minicluster-data/master-1/wal
uuid: "e8be870d7089424d82f5be0171ba5242"
format_stamp: "Formatted at 2026-08-12 05:20:45 on dist-test-slave-42z9"
server_key: "f837a7bd167f164bae5ce9ac1c4f2604"
server_key_iv: "c82ec8da91b1eed7a44e69b1c4c4d483"
server_key_version: "encryptionkey@0"
I20260812 05:20:45.848696 10473 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P e8be870d7089424d82f5be0171ba5242: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 05:20:45.959444 10473 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P e8be870d7089424d82f5be0171ba5242: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260812 05:20:45.959564 10473 log.cc:1196] T 00000000000000000000000000000000 P e8be870d7089424d82f5be0171ba5242: Deleting WAL directory at /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786512017613366-32144-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260812 05:20:45.960139 10473 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P e8be870d7089424d82f5be0171ba5242: Deleting consensus metadata
I20260812 05:20:45.960268 10473 master_runner.cc:315] Copying system tablet from 127.31.100.62:35003
I20260812 05:20:45.960865 10473 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P e8be870d7089424d82f5be0171ba5242: tablet copy: Beginning tablet copy session from remote peer at address 127.31.100.62:35003
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.31.100.60@KRBTEST.COM" created.
I20260812 05:20:45.967115 10651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:45.961216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.62:35003 (local address 127.0.0.1:48194)
0812 05:20:45.961396 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0812 05:20:45.961406 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0812 05:20:45.961519 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:20:45.961666 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:20:45.961667 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:20:45.961828 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0812 05:20:45.961875 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:20:45.961877 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:20:45.962741 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:20:45.962743 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:20:45.963381 (+ 638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:20:45.963385 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:20:45.963731 (+ 346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:20:45.963732 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:20:45.963785 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:45.964235 (+ 450us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 05:20:45.964240 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0812 05:20:45.964549 (+ 309us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 05:20:45.966053 (+ 1504us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 05:20:45.966054 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 05:20:45.966055 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 05:20:45.966260 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 05:20:45.966504 (+ 244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 05:20:45.966505 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 05:20:45.966505 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 05:20:45.966569 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 05:20:45.966770 (+ 201us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 05:20:45.966770 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 05:20:45.966816 (+ 46us) client_negotiation.cc:769] Sending connection context
0812 05:20:45.966900 (+ 84us) client_negotiation.cc:240] Negotiation successful
0812 05:20:45.967014 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":93,"threads_started":1}
I20260812 05:20:45.967128 9802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:45.961284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48194 (local address 127.31.100.62:35003)
0812 05:20:45.961338 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0812 05:20:45.961339 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0812 05:20:45.961501 (+ 162us) server_negotiation.cc:407] Connection header received
0812 05:20:45.961586 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:20:45.961588 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:20:45.961601 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:20:45.961648 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:20:45.961940 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:45.962660 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:45.963462 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:45.963667 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:45.963727 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:45.964629 (+ 902us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 05:20:45.964631 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 05:20:45.964631 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 05:20:45.964639 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0812 05:20:45.965946 (+ 1307us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:20:45.966363 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:20:45.966364 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:20:45.966364 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:20:45.966446 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:20:45.966625 (+ 179us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:20:45.966626 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:20:45.966626 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:20:45.966723 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 05:20:45.966767 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0812 05:20:45.966952 (+ 185us) server_negotiation.cc:299] Negotiation successful
0812 05:20:45.967034 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260812 05:20:45.967473 9772 tablet_copy_service.cc:140] P 070ed78c702d4333919d5d31da7b7608: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer e8be870d7089424d82f5be0171ba5242 ({username='oryx', principal='oryx/127.31.100.61@KRBTEST.COM'} at 127.0.0.1:48194)
I20260812 05:20:45.967578 9772 tablet_copy_service.cc:161] P 070ed78c702d4333919d5d31da7b7608: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer e8be870d7089424d82f5be0171ba5242 at {username='oryx', principal='oryx/127.31.100.61@KRBTEST.COM'} at 127.0.0.1:48194: session id = e8be870d7089424d82f5be0171ba5242-00000000000000000000000000000000
I20260812 05:20:45.968257 9772 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 070ed78c702d4333919d5d31da7b7608: Tablet Copy: opened 0 blocks and 1 log segments
I20260812 05:20:45.968811 10473 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260812 05:20:45.969901 10473 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P e8be870d7089424d82f5be0171ba5242: tablet copy: Starting download of 0 data blocks...
I20260812 05:20:45.970043 10473 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P e8be870d7089424d82f5be0171ba5242: tablet copy: Starting download of 1 WAL segments...
I20260812 05:20:45.970892 10473 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P e8be870d7089424d82f5be0171ba5242: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260812 05:20:46.063700 9772 tablet_copy_service.cc:342] P 070ed78c702d4333919d5d31da7b7608: Request end of tablet copy session e8be870d7089424d82f5be0171ba5242-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.31.100.61@KRBTEST.COM'} at 127.0.0.1:48194
I20260812 05:20:46.063789 9772 tablet_copy_service.cc:433] P 070ed78c702d4333919d5d31da7b7608: ending tablet copy session e8be870d7089424d82f5be0171ba5242-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer e8be870d7089424d82f5be0171ba5242
I20260812 05:20:46.067049 10473 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 05:20:46.068251 10658 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 05:20:46.068413 10653 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 05:20:46.068681 10473 server_base.cc:1034] running on GCE node
W20260812 05:20:46.069954 10655 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 05:20:46.070096 10473 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 05:20:46.070223 10473 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 05:20:46.071331 10473 hybrid_clock.cc:648] HybridClock initialized: now 1786512046071320 us; error 29 us; skew 500 ppm
I20260812 05:20:46.192777 10034 heartbeater.cc:503] Master 127.31.100.62:35003 was elected leader, sending a full tablet report...
Aug 12 05:20:46 dist-test-slave-42z9 krb5kdc[9665](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512046, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.100.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 05:20:46.269636 10473 init.cc:377] Logged in from keytab as oryx/127.31.100.61@KRBTEST.COM (short username oryx)
I20260812 05:20:46.269942 10473 webserver.cc:460] Webserver started at http://127.31.100.61:34501/ using document root <none> and password file <none>
I20260812 05:20:46.270049 10473 fs_manager.cc:359] Metadata directory not provided
I20260812 05:20:46.270076 10473 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 05:20:46.270677 10473 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.000s sys 0.000s
I20260812 05:20:46.271265 10672 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 05:20:46.271430 10473 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260812 05:20:46.271481 10473 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786512017613366-32144-0/minicluster-data/master-1/data,/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786512017613366-32144-0/minicluster-data/master-1/wal
uuid: "e8be870d7089424d82f5be0171ba5242"
format_stamp: "Formatted at 2026-08-12 05:20:45 on dist-test-slave-42z9"
server_key: "f837a7bd167f164bae5ce9ac1c4f2604"
server_key_iv: "c82ec8da91b1eed7a44e69b1c4c4d483"
server_key_version: "encryptionkey@0"
I20260812 05:20:46.271543 10473 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786512017613366-32144-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786512017613366-32144-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786512017613366-32144-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 05:20:46.292479 10473 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 05:20:46.293700 10473 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 05:20:46.298744 10473 rpc_server.cc:307] RPC server started. Bound to: 127.31.100.61:34209
I20260812 05:20:46.299665 10756 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.100.61:34209 every 8 connection(s)
I20260812 05:20:46.311625 10757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:46.306178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.61:34209 (local address 127.0.0.1:49854)
0812 05:20:46.306301 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0812 05:20:46.306312 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0812 05:20:46.306374 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:20:46.306556 (+ 182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:20:46.306557 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:20:46.306735 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0812 05:20:46.306783 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:20:46.306786 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:20:46.307661 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:20:46.307663 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:20:46.308181 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:20:46.308183 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:20:46.308544 (+ 361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:20:46.308545 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:20:46.308607 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:46.308983 (+ 376us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 05:20:46.308987 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0812 05:20:46.309297 (+ 310us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 05:20:46.310660 (+ 1363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 05:20:46.310661 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 05:20:46.310662 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 05:20:46.310831 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 05:20:46.311024 (+ 193us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 05:20:46.311025 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 05:20:46.311025 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 05:20:46.311079 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 05:20:46.311300 (+ 221us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 05:20:46.311300 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 05:20:46.311347 (+ 47us) client_negotiation.cc:769] Sending connection context
0812 05:20:46.311412 (+ 65us) client_negotiation.cc:240] Negotiation successful
0812 05:20:46.311523 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":72,"threads_started":1}
I20260812 05:20:46.311633 10758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:46.306247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49854 (local address 127.31.100.61:34209)
0812 05:20:46.306438 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0812 05:20:46.306440 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 05:20:46.306448 (+ 8us) server_negotiation.cc:407] Connection header received
0812 05:20:46.306472 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:20:46.306473 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:20:46.306482 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:20:46.306541 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:20:46.306855 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:46.307578 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:46.308254 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:46.308478 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:46.308533 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:46.309387 (+ 854us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 05:20:46.309389 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 05:20:46.309389 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 05:20:46.309396 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 05:20:46.310584 (+ 1188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:20:46.310892 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:20:46.310893 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:20:46.310894 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:20:46.310954 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:20:46.311130 (+ 176us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:20:46.311131 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:20:46.311131 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:20:46.311241 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 05:20:46.311300 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0812 05:20:46.311443 (+ 143us) server_negotiation.cc:299] Negotiation successful
0812 05:20:46.311523 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":93,"threads_started":1}
I20260812 05:20:46.355876 10256 heartbeater.cc:503] Master 127.31.100.62:35003 was elected leader, sending a full tablet report...
I20260812 05:20:46.364707 10473 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786512017613366-32144-0/minicluster-data/master-1/data/info.pb
I20260812 05:20:46.365674 10759 sys_catalog.cc:263] Verifying existing consensus state
I20260812 05:20:46.382750 10759 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e8be870d7089424d82f5be0171ba5242: Bootstrap starting.
I20260812 05:20:46.385119 10759 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e8be870d7089424d82f5be0171ba5242: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 05:20:46.385310 10759 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e8be870d7089424d82f5be0171ba5242: Bootstrap complete.
I20260812 05:20:46.385525 10759 raft_consensus.cc:348] T 00000000000000000000000000000000 P e8be870d7089424d82f5be0171ba5242 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "070ed78c702d4333919d5d31da7b7608" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 35003 } } peers { permanent_uuid: "e8be870d7089424d82f5be0171ba5242" member_type: NON_VOTER last_known_addr { host: "127.31.100.61" port: 34209 } attrs { promote: true } }
I20260812 05:20:46.385601 10759 raft_consensus.cc:729] T 00000000000000000000000000000000 P e8be870d7089424d82f5be0171ba5242 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e8be870d7089424d82f5be0171ba5242, State: Initialized, Role: LEARNER
I20260812 05:20:46.385639 10759 consensus_queue.cc:260] T 00000000000000000000000000000000 P e8be870d7089424d82f5be0171ba5242 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "070ed78c702d4333919d5d31da7b7608" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 35003 } } peers { permanent_uuid: "e8be870d7089424d82f5be0171ba5242" member_type: NON_VOTER last_known_addr { host: "127.31.100.61" port: 34209 } attrs { promote: true } }
I20260812 05:20:46.385819 10759 sys_catalog.cc:564] T 00000000000000000000000000000000 P e8be870d7089424d82f5be0171ba5242 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 05:20:46.385825 10762 sys_catalog.cc:455] T 00000000000000000000000000000000 P e8be870d7089424d82f5be0171ba5242 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "070ed78c702d4333919d5d31da7b7608" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 35003 } } peers { permanent_uuid: "e8be870d7089424d82f5be0171ba5242" member_type: NON_VOTER last_known_addr { host: "127.31.100.61" port: 34209 } attrs { promote: true } } }
I20260812 05:20:46.385958 10762 sys_catalog.cc:458] T 00000000000000000000000000000000 P e8be870d7089424d82f5be0171ba5242 [sys.catalog]: This master's current role is: LEARNER
I20260812 05:20:46.387123 10783 catalog_manager.cc:1259] Loaded cluster ID: 1c12969b4cd344eab17b8b9fb1e66566
I20260812 05:20:46.387169 10783 catalog_manager.cc:1552] T 00000000000000000000000000000000 P e8be870d7089424d82f5be0171ba5242: loading cluster ID for follower catalog manager: success
I20260812 05:20:46.388175 10783 catalog_manager.cc:1574] T 00000000000000000000000000000000 P e8be870d7089424d82f5be0171ba5242: acquiring CA information for follower catalog manager: success
I20260812 05:20:46.388422 10783 catalog_manager.cc:1602] T 00000000000000000000000000000000 P e8be870d7089424d82f5be0171ba5242: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260812 05:20:46.509521 10466 heartbeater.cc:503] Master 127.31.100.62:35003 was elected leader, sending a full tablet report...
I20260812 05:20:46.709556 10716 raft_consensus.cc:1203] T 00000000000000000000000000000000 P e8be870d7089424d82f5be0171ba5242 [term 1 LEARNER]: Deduplicated request from leader. Original: 0.0->[1.1-1.5] Dedup: 1.5->[]
I20260812 05:20:46.709762 10762 sys_catalog.cc:455] T 00000000000000000000000000000000 P e8be870d7089424d82f5be0171ba5242 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 070ed78c702d4333919d5d31da7b7608. Latest consensus state: current_term: 1 leader_uuid: "070ed78c702d4333919d5d31da7b7608" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "070ed78c702d4333919d5d31da7b7608" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 35003 } } peers { permanent_uuid: "e8be870d7089424d82f5be0171ba5242" member_type: NON_VOTER last_known_addr { host: "127.31.100.61" port: 34209 } attrs { promote: true } } }
I20260812 05:20:46.709889 10762 sys_catalog.cc:458] T 00000000000000000000000000000000 P e8be870d7089424d82f5be0171ba5242 [sys.catalog]: This master's current role is: LEARNER
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.31.100.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786512017613366-32144-0/krb5kdc/oryx_127.31.100.60.keytab.
Entry for principal oryx/127.31.100.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786512017613366-32144-0/krb5kdc/oryx_127.31.100.60.keytab.
W20260812 05:20:46.803576 32144 mini_kdc.cc:262] Time spent creating service keytab for oryx/127.31.100.60: real 0.975s user 0.000s sys 0.006s
I20260812 05:20:46.803922 32144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0waFvT/build/release/bin/kudu
/tmp/dist-test-task0waFvT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786512017613366-32144-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786512017613366-32144-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786512017613366-32144-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786512017613366-32144-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.100.60:37129
--webserver_interface=127.31.100.60
--webserver_port=0
--keytab_file=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786512017613366-32144-0/krb5kdc/oryx_127.31.100.60.keytab
--principal=oryx/127.31.100.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.100.20:44403
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.100.62:35003,127.31.100.61:34209,127.31.100.60:37129
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786512017613366-32144-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786512017613366-32144-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786512017613366-32144-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 05:20:46.885360 10785 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 05:20:46.885617 10785 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 05:20:46.885666 10785 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 05:20:46.886832 10785 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 05:20:46.886873 10785 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 05:20:46.886891 10785 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 05:20:46.886906 10785 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 05:20:46.886947 10785 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 05:20:46.888449 10785 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.100.20:44403
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786512017613366-32144-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786512017613366-32144-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.31.100.62:35003,127.31.100.61:34209,127.31.100.60:37129
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786512017613366-32144-0/krb5kdc/oryx_127.31.100.60.keytab
--principal=oryx/127.31.100.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.100.60:37129
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786512017613366-32144-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.31.100.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10785
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786512017613366-32144-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 04:20:34 UTC on ccycloud.quasar-phqdqp.root.comops.site
build id 4581871
I20260812 05:20:46.888698 10785 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 05:20:46.888897 10785 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 05:20:46.893344 10794 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 05:20:46.893416 10792 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 05:20:46.893373 10791 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 05:20:46.893549 10785 server_base.cc:1034] running on GCE node
I20260812 05:20:46.893774 10785 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 05:20:46.893998 10785 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 05:20:46.895133 10785 hybrid_clock.cc:648] HybridClock initialized: now 1786512046895130 us; error 32 us; skew 500 ppm
Aug 12 05:20:46 dist-test-slave-42z9 krb5kdc[9665](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512046, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.100.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 05:20:46.897552 10785 init.cc:377] Logged in from keytab as oryx/127.31.100.60@KRBTEST.COM (short username oryx)
I20260812 05:20:46.897974 10785 webserver.cc:460] Webserver started at http://127.31.100.60:46177/ using document root <none> and password file <none>
I20260812 05:20:46.898187 10785 fs_manager.cc:359] Metadata directory not provided
I20260812 05:20:46.898232 10785 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 05:20:46.898361 10785 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 05:20:46.899355 10785 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786512017613366-32144-0/minicluster-data/master-2/data/instance:
uuid: "22c6383cda19494f95cfee62ca4d0ccf"
format_stamp: "Formatted at 2026-08-12 05:20:46 on dist-test-slave-42z9"
server_key: "8f263171724ab9ac250a773f65523438"
server_key_iv: "3dba32fda7cef315846a6896c4522068"
server_key_version: "encryptionkey@0"
I20260812 05:20:46.899689 10785 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786512017613366-32144-0/minicluster-data/master-2/wal/instance:
uuid: "22c6383cda19494f95cfee62ca4d0ccf"
format_stamp: "Formatted at 2026-08-12 05:20:46 on dist-test-slave-42z9"
server_key: "8f263171724ab9ac250a773f65523438"
server_key_iv: "3dba32fda7cef315846a6896c4522068"
server_key_version: "encryptionkey@0"
I20260812 05:20:46.901024 10785 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260812 05:20:46.901748 10806 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 05:20:46.901921 10785 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260812 05:20:46.902017 10785 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786512017613366-32144-0/minicluster-data/master-2/data,/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786512017613366-32144-0/minicluster-data/master-2/wal
uuid: "22c6383cda19494f95cfee62ca4d0ccf"
format_stamp: "Formatted at 2026-08-12 05:20:46 on dist-test-slave-42z9"
server_key: "8f263171724ab9ac250a773f65523438"
server_key_iv: "3dba32fda7cef315846a6896c4522068"
server_key_version: "encryptionkey@0"
I20260812 05:20:46.902083 10785 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786512017613366-32144-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786512017613366-32144-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786512017613366-32144-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 05:20:46.914309 10785 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 05:20:46.916272 10785 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 05:20:46.916394 10785 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 05:20:46.920372 10785 rpc_server.cc:307] RPC server started. Bound to: 127.31.100.60:37129
I20260812 05:20:46.920413 10879 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.100.60:37129 every 8 connection(s)
I20260812 05:20:46.920719 10785 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786512017613366-32144-0/minicluster-data/master-2/data/info.pb
I20260812 05:20:46.921689 10880 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 05:20:46.923146 10880 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.100.62" port: 35003 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 05:20:46 dist-test-slave-42z9 krb5kdc[9665](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512046, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.100.60@KRBTEST.COM for oryx/127.31.100.62@KRBTEST.COM
I20260812 05:20:46.929422 32144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0waFvT/build/release/bin/kudu as pid 10785
I20260812 05:20:46.931409 10887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:46.923840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48206 (local address 127.31.100.62:35003)
0812 05:20:46.923953 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0812 05:20:46.923955 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 05:20:46.924010 (+ 55us) server_negotiation.cc:407] Connection header received
0812 05:20:46.924473 (+ 463us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:20:46.924474 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:20:46.924485 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:20:46.924552 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:20:46.925071 (+ 519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:46.925634 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:46.926528 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:46.926729 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:46.926794 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:46.928655 (+ 1861us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 05:20:46.928657 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 05:20:46.928657 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 05:20:46.928663 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0812 05:20:46.930052 (+ 1389us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:20:46.930517 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:20:46.930518 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:20:46.930519 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:20:46.930579 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:20:46.930830 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:20:46.930830 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:20:46.930831 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:20:46.930932 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 05:20:46.930994 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0812 05:20:46.931220 (+ 226us) server_negotiation.cc:299] Negotiation successful
0812 05:20:46.931308 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92,"thread_start_us":61,"threads_started":1}
I20260812 05:20:46.931445 10883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:46.923748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.62:35003 (local address 127.0.0.1:48206)
0812 05:20:46.923901 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0812 05:20:46.923940 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0812 05:20:46.924398 (+ 458us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:20:46.924603 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:20:46.924606 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:20:46.924821 (+ 215us) client_negotiation.cc:189] Negotiated authn=SASL
0812 05:20:46.925000 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:20:46.925012 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:20:46.925696 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:20:46.925697 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:20:46.926453 (+ 756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:20:46.926456 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:20:46.926811 (+ 355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:20:46.926812 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:20:46.926902 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:46.927311 (+ 409us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 05:20:46.927318 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0812 05:20:46.928576 (+ 1258us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 05:20:46.930145 (+ 1569us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 05:20:46.930151 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 05:20:46.930153 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 05:20:46.930445 (+ 292us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 05:20:46.930638 (+ 193us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 05:20:46.930639 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 05:20:46.930639 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 05:20:46.930772 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 05:20:46.930991 (+ 219us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 05:20:46.930996 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 05:20:46.931103 (+ 107us) client_negotiation.cc:769] Sending connection context
0812 05:20:46.931181 (+ 78us) client_negotiation.cc:240] Negotiation successful
0812 05:20:46.931291 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":68,"threads_started":1}
I20260812 05:20:46.931902 10880 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.100.61" port: 34209 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 05:20:46 dist-test-slave-42z9 krb5kdc[9665](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512044, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.31.100.60@KRBTEST.COM
Aug 12 05:20:46 dist-test-slave-42z9 krb5kdc[9665](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512046, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.100.60@KRBTEST.COM for oryx/127.31.100.61@KRBTEST.COM
I20260812 05:20:46.938449 10883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:46.932117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.61:34209 (local address 127.0.0.1:49858)
0812 05:20:46.932154 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0812 05:20:46.932165 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0812 05:20:46.932199 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:20:46.932469 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:20:46.932470 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:20:46.932574 (+ 104us) client_negotiation.cc:189] Negotiated authn=SASL
0812 05:20:46.932616 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:20:46.932617 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:20:46.933353 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:20:46.933354 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:20:46.933914 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:20:46.933917 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:20:46.934408 (+ 491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:20:46.934409 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:20:46.934465 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:46.934761 (+ 296us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 05:20:46.934766 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0812 05:20:46.935888 (+ 1122us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 05:20:46.937472 (+ 1584us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 05:20:46.937473 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 05:20:46.937473 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 05:20:46.937658 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 05:20:46.937841 (+ 183us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 05:20:46.937842 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 05:20:46.937842 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 05:20:46.937930 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 05:20:46.938169 (+ 239us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 05:20:46.938169 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 05:20:46.938211 (+ 42us) client_negotiation.cc:769] Sending connection context
0812 05:20:46.938275 (+ 64us) client_negotiation.cc:240] Negotiation successful
0812 05:20:46.938363 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23}
I20260812 05:20:46.938551 10892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:46.929971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55744 (local address 127.31.100.60:37129)
0812 05:20:46.930105 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0812 05:20:46.930109 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 05:20:46.930124 (+ 15us) server_negotiation.cc:407] Connection header received
0812 05:20:46.930184 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:20:46.930186 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:20:46.930196 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:20:46.930252 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:20:46.930944 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:46.931865 (+ 921us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:46.932750 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:46.933021 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:46.933067 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:46.935615 (+ 2548us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 05:20:46.935624 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 05:20:46.935624 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 05:20:46.935631 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 05:20:46.937206 (+ 1575us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:20:46.937706 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:20:46.937709 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:20:46.937709 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:20:46.937782 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:20:46.937979 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:20:46.937980 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:20:46.937980 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:20:46.938140 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 05:20:46.938261 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0812 05:20:46.938365 (+ 104us) server_negotiation.cc:299] Negotiation successful
0812 05:20:46.938433 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106,"thread_start_us":83,"threads_started":1}
I20260812 05:20:46.938481 10893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:46.932200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49858 (local address 127.31.100.61:34209)
0812 05:20:46.932339 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0812 05:20:46.932342 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 05:20:46.932350 (+ 8us) server_negotiation.cc:407] Connection header received
0812 05:20:46.932378 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:20:46.932378 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:20:46.932394 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:20:46.932462 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:20:46.932674 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:46.933265 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:46.933989 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:46.934329 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:46.934401 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:46.935983 (+ 1582us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 05:20:46.935984 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 05:20:46.935984 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 05:20:46.935990 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0812 05:20:46.937355 (+ 1365us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:20:46.937714 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:20:46.937715 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:20:46.937715 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:20:46.937782 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:20:46.937982 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:20:46.937983 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:20:46.937983 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:20:46.938103 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 05:20:46.938171 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0812 05:20:46.938308 (+ 137us) server_negotiation.cc:299] Negotiation successful
0812 05:20:46.938389 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":72,"threads_started":1}
I20260812 05:20:46.939153 10880 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.100.60" port: 37129 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 05:20:46 dist-test-slave-42z9 krb5kdc[9665](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512046, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.100.60@KRBTEST.COM for oryx/127.31.100.60@KRBTEST.COM
I20260812 05:20:46.944240 10883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:46.939372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.60:37129 (local address 127.0.0.1:55750)
0812 05:20:46.939442 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0812 05:20:46.939449 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0812 05:20:46.939485 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:20:46.939573 (+ 88us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:20:46.939573 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:20:46.939671 (+ 98us) client_negotiation.cc:189] Negotiated authn=SASL
0812 05:20:46.939695 (+ 24us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:20:46.939696 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:20:46.940213 (+ 517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:20:46.940213 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:20:46.940642 (+ 429us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:20:46.940642 (+ 0us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:20:46.940906 (+ 264us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:20:46.940907 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:20:46.940938 (+ 31us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:46.941210 (+ 272us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 05:20:46.941214 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0812 05:20:46.942249 (+ 1035us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 05:20:46.943583 (+ 1334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 05:20:46.943583 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 05:20:46.943583 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 05:20:46.943716 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 05:20:46.943838 (+ 122us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 05:20:46.943839 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 05:20:46.943839 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 05:20:46.943877 (+ 38us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 05:20:46.944036 (+ 159us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 05:20:46.944037 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 05:20:46.944065 (+ 28us) client_negotiation.cc:769] Sending connection context
0812 05:20:46.944107 (+ 42us) client_negotiation.cc:240] Negotiation successful
0812 05:20:46.944170 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59}
I20260812 05:20:46.944289 10892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:46.939413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55750 (local address 127.31.100.60:37129)
0812 05:20:46.939443 (+ 30us) server_negotiation.cc:206] Beginning negotiation
0812 05:20:46.939444 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0812 05:20:46.939483 (+ 39us) server_negotiation.cc:407] Connection header received
0812 05:20:46.939525 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:20:46.939526 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:20:46.939534 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:20:46.939571 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:20:46.939729 (+ 158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:46.940165 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:46.940685 (+ 520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:46.940860 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:46.940905 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:46.942318 (+ 1413us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 05:20:46.942319 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 05:20:46.942319 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 05:20:46.942325 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0812 05:20:46.943523 (+ 1198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:20:46.943760 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:20:46.943761 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:20:46.943762 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:20:46.943801 (+ 39us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:20:46.943909 (+ 108us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:20:46.943910 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:20:46.943910 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:20:46.943997 (+ 87us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 05:20:46.944037 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0812 05:20:46.944122 (+ 85us) server_negotiation.cc:299] Negotiation successful
0812 05:20:46.944209 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":35,"server-negotiator.queue_time_us":15}
I20260812 05:20:46.945724 10880 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 22c6383cda19494f95cfee62ca4d0ccf: Bootstrap starting.
I20260812 05:20:46.946523 10880 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 22c6383cda19494f95cfee62ca4d0ccf: Neither blocks nor log segments found. Creating new log.
I20260812 05:20:46.946766 10880 log.cc:824] T 00000000000000000000000000000000 P 22c6383cda19494f95cfee62ca4d0ccf: Log is configured to *not* fsync() on all Append() calls
I20260812 05:20:46.947444 10880 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 22c6383cda19494f95cfee62ca4d0ccf: No bootstrap required, opened a new log
I20260812 05:20:46.948819 10880 raft_consensus.cc:348] T 00000000000000000000000000000000 P 22c6383cda19494f95cfee62ca4d0ccf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "070ed78c702d4333919d5d31da7b7608" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 35003 } } peers { permanent_uuid: "e8be870d7089424d82f5be0171ba5242" member_type: VOTER last_known_addr { host: "127.31.100.61" port: 34209 } } peers { permanent_uuid: "22c6383cda19494f95cfee62ca4d0ccf" member_type: VOTER last_known_addr { host: "127.31.100.60" port: 37129 } }
I20260812 05:20:46.948936 10880 raft_consensus.cc:374] T 00000000000000000000000000000000 P 22c6383cda19494f95cfee62ca4d0ccf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 05:20:46.948968 10880 raft_consensus.cc:729] T 00000000000000000000000000000000 P 22c6383cda19494f95cfee62ca4d0ccf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22c6383cda19494f95cfee62ca4d0ccf, State: Initialized, Role: FOLLOWER
I20260812 05:20:46.949085 10880 consensus_queue.cc:260] T 00000000000000000000000000000000 P 22c6383cda19494f95cfee62ca4d0ccf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "070ed78c702d4333919d5d31da7b7608" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 35003 } } peers { permanent_uuid: "e8be870d7089424d82f5be0171ba5242" member_type: VOTER last_known_addr { host: "127.31.100.61" port: 34209 } } peers { permanent_uuid: "22c6383cda19494f95cfee62ca4d0ccf" member_type: VOTER last_known_addr { host: "127.31.100.60" port: 37129 } }
I20260812 05:20:46.949368 10880 sys_catalog.cc:564] T 00000000000000000000000000000000 P 22c6383cda19494f95cfee62ca4d0ccf [sys.catalog]: configured and running, proceeding with master startup.
I20260812 05:20:46.949333 10897 sys_catalog.cc:455] T 00000000000000000000000000000000 P 22c6383cda19494f95cfee62ca4d0ccf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "070ed78c702d4333919d5d31da7b7608" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 35003 } } peers { permanent_uuid: "e8be870d7089424d82f5be0171ba5242" member_type: VOTER last_known_addr { host: "127.31.100.61" port: 34209 } } peers { permanent_uuid: "22c6383cda19494f95cfee62ca4d0ccf" member_type: VOTER last_known_addr { host: "127.31.100.60" port: 37129 } } }
I20260812 05:20:46.949434 10897 sys_catalog.cc:458] T 00000000000000000000000000000000 P 22c6383cda19494f95cfee62ca4d0ccf [sys.catalog]: This master's current role is: FOLLOWER
W20260812 05:20:46.951090 10916 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 22c6383cda19494f95cfee62ca4d0ccf: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 05:20:46.951150 10916 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 05:20:46.953190 10785 master_runner.cc:428] Detected that this master 22c6383cda19494f95cfee62ca4d0ccf is joining an existing cluster
I20260812 05:20:46.953229 10785 master_runner.cc:432] Initiating AddMaster RPC to add 127.31.100.60:37129
I20260812 05:20:46.961127 10929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:46.954788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.60:37129 (local address 127.0.0.1:55754)
0812 05:20:46.954978 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0812 05:20:46.954987 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0812 05:20:46.955046 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:20:46.955176 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:20:46.955176 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:20:46.955407 (+ 231us) client_negotiation.cc:189] Negotiated authn=SASL
0812 05:20:46.955461 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:20:46.955463 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:20:46.956274 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:20:46.956275 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:20:46.957129 (+ 854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:20:46.957132 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:20:46.957629 (+ 497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:20:46.957630 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:20:46.957679 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:46.958186 (+ 507us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 05:20:46.958190 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0812 05:20:46.958542 (+ 352us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 05:20:46.960072 (+ 1530us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 05:20:46.960073 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 05:20:46.960073 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 05:20:46.960238 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 05:20:46.960435 (+ 197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 05:20:46.960435 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 05:20:46.960435 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 05:20:46.960495 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 05:20:46.960738 (+ 243us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 05:20:46.960739 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 05:20:46.960789 (+ 50us) client_negotiation.cc:769] Sending connection context
0812 05:20:46.960830 (+ 41us) client_negotiation.cc:240] Negotiation successful
0812 05:20:46.961044 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171,"mutex_wait_us":30,"thread_start_us":69,"threads_started":1}
I20260812 05:20:46.961242 10892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:46.954900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55754 (local address 127.31.100.60:37129)
0812 05:20:46.954933 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0812 05:20:46.954934 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0812 05:20:46.955029 (+ 95us) server_negotiation.cc:407] Connection header received
0812 05:20:46.955101 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:20:46.955101 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:20:46.955108 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:20:46.955132 (+ 24us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:20:46.955542 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:46.956204 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:46.957211 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:46.957516 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:46.957553 (+ 37us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:46.958622 (+ 1069us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 05:20:46.958623 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 05:20:46.958623 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 05:20:46.958630 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 05:20:46.959994 (+ 1364us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:20:46.960302 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:20:46.960303 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:20:46.960303 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:20:46.960360 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:20:46.960548 (+ 188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:20:46.960549 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:20:46.960549 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:20:46.960681 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 05:20:46.960752 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0812 05:20:46.960865 (+ 113us) server_negotiation.cc:299] Negotiation successful
0812 05:20:46.961158 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":184,"server-negotiator.queue_time_us":18}
I20260812 05:20:46.981338 10931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:46.954607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.61:34209 (local address 127.0.0.1:49862)
0812 05:20:46.954905 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0812 05:20:46.954914 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0812 05:20:46.954991 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:20:46.955157 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:20:46.955158 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:20:46.955360 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0812 05:20:46.955437 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:20:46.955440 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:20:46.956375 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:20:46.956376 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:20:46.957206 (+ 830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:20:46.957209 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:20:46.957769 (+ 560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:20:46.957770 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:20:46.957849 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:46.978261 (+ 20412us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 05:20:46.978268 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0812 05:20:46.978756 (+ 488us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 05:20:46.980318 (+ 1562us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 05:20:46.980319 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 05:20:46.980319 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 05:20:46.980500 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 05:20:46.980677 (+ 177us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 05:20:46.980678 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 05:20:46.980678 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 05:20:46.980730 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 05:20:46.980988 (+ 258us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 05:20:46.980989 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 05:20:46.981039 (+ 50us) client_negotiation.cc:769] Sending connection context
0812 05:20:46.981107 (+ 68us) client_negotiation.cc:240] Negotiation successful
0812 05:20:46.981216 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":275,"mutex_wait_us":1,"thread_start_us":84,"threads_started":1}
I20260812 05:20:46.981325 10893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:46.954675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49862 (local address 127.31.100.61:34209)
0812 05:20:46.954714 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0812 05:20:46.954716 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 05:20:46.954967 (+ 251us) server_negotiation.cc:407] Connection header received
0812 05:20:46.955057 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:20:46.955059 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:20:46.955071 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:20:46.955100 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:20:46.955532 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:46.956279 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:46.957303 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:46.957648 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:46.957692 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:46.978880 (+ 21188us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 05:20:46.978882 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 05:20:46.978883 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 05:20:46.978891 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0812 05:20:46.980239 (+ 1348us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:20:46.980569 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:20:46.980570 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:20:46.980571 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:20:46.980624 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:20:46.980781 (+ 157us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:20:46.980781 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:20:46.980782 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:20:46.980918 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 05:20:46.980991 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0812 05:20:46.981140 (+ 149us) server_negotiation.cc:299] Negotiation successful
0812 05:20:46.981236 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260812 05:20:46.981993 10887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:46.954206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48212 (local address 127.31.100.62:35003)
0812 05:20:46.954257 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0812 05:20:46.954259 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 05:20:46.954601 (+ 342us) server_negotiation.cc:407] Connection header received
0812 05:20:46.954702 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:20:46.954703 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:20:46.954718 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:20:46.954747 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:20:46.955193 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:46.956142 (+ 949us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:46.957324 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:46.957637 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:46.957674 (+ 37us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:46.978985 (+ 21311us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 05:20:46.978987 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 05:20:46.978988 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 05:20:46.978997 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0812 05:20:46.980517 (+ 1520us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:20:46.980889 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:20:46.980890 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:20:46.980890 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:20:46.980948 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:20:46.981141 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:20:46.981142 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:20:46.981142 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:20:46.981273 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 05:20:46.981316 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0812 05:20:46.981823 (+ 507us) server_negotiation.cc:299] Negotiation successful
0812 05:20:46.981924 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260812 05:20:46.982287 10928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:46.954402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.62:35003 (local address 127.0.0.1:48212)
0812 05:20:46.954542 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0812 05:20:46.954553 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0812 05:20:46.954628 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:20:46.954815 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:20:46.954815 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:20:46.955033 (+ 218us) client_negotiation.cc:189] Negotiated authn=SASL
0812 05:20:46.955108 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:20:46.955110 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:20:46.956230 (+ 1120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:20:46.956231 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:20:46.957249 (+ 1018us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:20:46.957251 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:20:46.957743 (+ 492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:20:46.957744 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:20:46.957800 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:46.978475 (+ 20675us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 05:20:46.978481 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0812 05:20:46.978809 (+ 328us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 05:20:46.980612 (+ 1803us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 05:20:46.980613 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 05:20:46.980614 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 05:20:46.980812 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 05:20:46.981021 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 05:20:46.981022 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 05:20:46.981022 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 05:20:46.981073 (+ 51us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 05:20:46.981652 (+ 579us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 05:20:46.981652 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 05:20:46.981697 (+ 45us) client_negotiation.cc:769] Sending connection context
0812 05:20:46.982072 (+ 375us) client_negotiation.cc:240] Negotiation successful
0812 05:20:46.982187 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"mutex_wait_us":37,"thread_start_us":76,"threads_started":1}
Aug 12 05:20:46 dist-test-slave-42z9 krb5kdc[9665](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512044, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.100.62@KRBTEST.COM for oryx/127.31.100.60@KRBTEST.COM
I20260812 05:20:46.989826 32144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0waFvT/build/release/bin/kudu with pid 9830
I20260812 05:20:46.990801 10892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:46.984371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55756 (local address 127.31.100.60:37129)
0812 05:20:46.984413 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0812 05:20:46.984414 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0812 05:20:46.984467 (+ 53us) server_negotiation.cc:407] Connection header received
0812 05:20:46.984553 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:20:46.984553 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:20:46.984561 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:20:46.984602 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:20:46.984904 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:46.985335 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:46.986338 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:46.986527 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:46.986607 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:46.988479 (+ 1872us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 05:20:46.988480 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 05:20:46.988481 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 05:20:46.988486 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0812 05:20:46.989832 (+ 1346us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:20:46.990197 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:20:46.990198 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:20:46.990198 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:20:46.990243 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:20:46.990388 (+ 145us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:20:46.990388 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:20:46.990388 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:20:46.990492 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 05:20:46.990522 (+ 30us) server_negotiation.cc:1035] Waiting for connection context
0812 05:20:46.990672 (+ 150us) server_negotiation.cc:299] Negotiation successful
0812 05:20:46.990734 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260812 05:20:46.991201 10932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:46.984292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.60:37129 (local address 127.0.0.1:55756)
0812 05:20:46.984413 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0812 05:20:46.984423 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0812 05:20:46.984484 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:20:46.984607 (+ 123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:20:46.984608 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:20:46.984787 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0812 05:20:46.984837 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:20:46.984839 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:20:46.985395 (+ 556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:20:46.985396 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:20:46.986268 (+ 872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:20:46.986271 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:20:46.986638 (+ 367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:20:46.986639 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:20:46.986706 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:46.987114 (+ 408us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 05:20:46.987118 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0812 05:20:46.988397 (+ 1279us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 05:20:46.989948 (+ 1551us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 05:20:46.989950 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 05:20:46.989950 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 05:20:46.990128 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 05:20:46.990296 (+ 168us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 05:20:46.990296 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 05:20:46.990297 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 05:20:46.990340 (+ 43us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 05:20:46.990558 (+ 218us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 05:20:46.990558 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 05:20:46.990593 (+ 35us) client_negotiation.cc:769] Sending connection context
0812 05:20:46.991034 (+ 441us) client_negotiation.cc:240] Negotiation successful
0812 05:20:46.991126 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":79,"threads_started":1}
I20260812 05:20:46.993669 9740 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.31.100.60:37129: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "22c6383cda19494f95cfee62ca4d0ccf"
member_type: NON_VOTER
last_known_addr {
host: "127.31.100.60"
port: 37129
}
attrs {
promote: true
}
}
dest_uuid: "070ed78c702d4333919d5d31da7b7608"
cas_config_opid_index: 5
I20260812 05:20:46.993919 9740 consensus_queue.cc:237] T 00000000000000000000000000000000 P 070ed78c702d4333919d5d31da7b7608 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "070ed78c702d4333919d5d31da7b7608" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 35003 } } peers { permanent_uuid: "e8be870d7089424d82f5be0171ba5242" member_type: NON_VOTER last_known_addr { host: "127.31.100.61" port: 34209 } attrs { promote: true } } peers { permanent_uuid: "22c6383cda19494f95cfee62ca4d0ccf" member_type: NON_VOTER last_known_addr { host: "127.31.100.60" port: 37129 } attrs { promote: true } }
I20260812 05:20:46.994515 10716 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e8be870d7089424d82f5be0171ba5242 [term 1 LEARNER]: Refusing update from remote peer 070ed78c702d4333919d5d31da7b7608: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260812 05:20:46.994561 10845 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 22c6383cda19494f95cfee62ca4d0ccf [term 0 FOLLOWER]: Advancing to term 1
I20260812 05:20:46.994730 10632 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 070ed78c702d4333919d5d31da7b7608 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 22c6383cda19494f95cfee62ca4d0ccf (127.31.100.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "070ed78c702d4333919d5d31da7b7608" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 35003 } } peers { permanent_uuid: "e8be870d7089424d82f5be0171ba5242" member_type: NON_VOTER last_known_addr { host: "127.31.100.61" port: 34209 } attrs { promote: true } } peers { permanent_uuid: "22c6383cda19494f95cfee62ca4d0ccf" member_type: NON_VOTER last_known_addr { host: "127.31.100.60" port: 37129 } attrs { promote: true } } }
I20260812 05:20:46.997668 10936 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 070ed78c702d4333919d5d31da7b7608 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8be870d7089424d82f5be0171ba5242" member_type: NON_VOTER last_known_addr { host: "127.31.100.61" port: 34209 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260812 05:20:46.998211 10716 raft_consensus.cc:2932] T 00000000000000000000000000000000 P e8be870d7089424d82f5be0171ba5242 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 22c6383cda19494f95cfee62ca4d0ccf (127.31.100.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "070ed78c702d4333919d5d31da7b7608" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 35003 } } peers { permanent_uuid: "e8be870d7089424d82f5be0171ba5242" member_type: NON_VOTER last_known_addr { host: "127.31.100.61" port: 34209 } attrs { promote: true } } peers { permanent_uuid: "22c6383cda19494f95cfee62ca4d0ccf" member_type: NON_VOTER last_known_addr { host: "127.31.100.60" port: 37129 } attrs { promote: true } } }
I20260812 05:20:47.000532 32144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0waFvT/build/release/bin/kudu
/tmp/dist-test-task0waFvT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786512017613366-32144-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786512017613366-32144-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786512017613366-32144-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786512017613366-32144-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.100.1:44127
--local_ip_for_outbound_sockets=127.31.100.1
--tserver_master_addrs=127.31.100.62:35003
--webserver_port=36363
--webserver_interface=127.31.100.1
--keytab_file=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786512017613366-32144-0/krb5kdc/oryx_127.31.100.1.keytab
--principal=oryx/127.31.100.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.100.20:44403
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.31.100.62:35003,127.31.100.61:34209
--tserver_master_addrs=127.31.100.62:35003,127.31.100.61:34209,127.31.100.60:37129 with env {KRB5CCNAME=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786512017613366-32144-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786512017613366-32144-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786512017613366-32144-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 05:20:47.075752 10845 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 22c6383cda19494f95cfee62ca4d0ccf [term 1 FOLLOWER]: Refusing update from remote peer 070ed78c702d4333919d5d31da7b7608: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260812 05:20:47.076447 10632 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.31.100.60:37129
I20260812 05:20:47.076563 10632 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 070ed78c702d4333919d5d31da7b7608 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22c6383cda19494f95cfee62ca4d0ccf" member_type: NON_VOTER last_known_addr { host: "127.31.100.60" port: 37129 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 05:20:47.076826 10632 sys_catalog.cc:455] T 00000000000000000000000000000000 P 070ed78c702d4333919d5d31da7b7608 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "070ed78c702d4333919d5d31da7b7608" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "070ed78c702d4333919d5d31da7b7608" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 35003 } } peers { permanent_uuid: "e8be870d7089424d82f5be0171ba5242" member_type: NON_VOTER last_known_addr { host: "127.31.100.61" port: 34209 } attrs { promote: true } } peers { permanent_uuid: "22c6383cda19494f95cfee62ca4d0ccf" member_type: NON_VOTER last_known_addr { host: "127.31.100.60" port: 37129 } attrs { promote: true } } }
I20260812 05:20:47.076895 10632 sys_catalog.cc:458] T 00000000000000000000000000000000 P 070ed78c702d4333919d5d31da7b7608 [sys.catalog]: This master's current role is: LEADER
I20260812 05:20:47.077278 10785 master.cc:537] Master@127.31.100.60:37129 shutting down...
I20260812 05:20:47.079521 10762 sys_catalog.cc:455] T 00000000000000000000000000000000 P e8be870d7089424d82f5be0171ba5242 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "070ed78c702d4333919d5d31da7b7608" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "070ed78c702d4333919d5d31da7b7608" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 35003 } } peers { permanent_uuid: "e8be870d7089424d82f5be0171ba5242" member_type: NON_VOTER last_known_addr { host: "127.31.100.61" port: 34209 } attrs { promote: true } } peers { permanent_uuid: "22c6383cda19494f95cfee62ca4d0ccf" member_type: NON_VOTER last_known_addr { host: "127.31.100.60" port: 37129 } attrs { promote: true } } }
I20260812 05:20:47.079619 10762 sys_catalog.cc:458] T 00000000000000000000000000000000 P e8be870d7089424d82f5be0171ba5242 [sys.catalog]: This master's current role is: LEARNER
I20260812 05:20:47.079965 10632 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 070ed78c702d4333919d5d31da7b7608: attempting to promote NON_VOTER e8be870d7089424d82f5be0171ba5242 to VOTER
I20260812 05:20:47.080143 10632 consensus_queue.cc:237] T 00000000000000000000000000000000 P 070ed78c702d4333919d5d31da7b7608 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "070ed78c702d4333919d5d31da7b7608" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 35003 } } peers { permanent_uuid: "e8be870d7089424d82f5be0171ba5242" member_type: VOTER last_known_addr { host: "127.31.100.61" port: 34209 } attrs { promote: false } } peers { permanent_uuid: "22c6383cda19494f95cfee62ca4d0ccf" member_type: NON_VOTER last_known_addr { host: "127.31.100.60" port: 37129 } attrs { promote: true } }
I20260812 05:20:47.080864 10716 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e8be870d7089424d82f5be0171ba5242 [term 1 LEARNER]: Refusing update from remote peer 070ed78c702d4333919d5d31da7b7608: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260812 05:20:47.081089 10632 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 070ed78c702d4333919d5d31da7b7608 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8be870d7089424d82f5be0171ba5242" member_type: VOTER last_known_addr { host: "127.31.100.61" port: 34209 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260812 05:20:47.081790 10632 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 070ed78c702d4333919d5d31da7b7608 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, e8be870d7089424d82f5be0171ba5242 (127.31.100.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "070ed78c702d4333919d5d31da7b7608" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 35003 } } peers { permanent_uuid: "e8be870d7089424d82f5be0171ba5242" member_type: VOTER last_known_addr { host: "127.31.100.61" port: 34209 } attrs { promote: false } } peers { permanent_uuid: "22c6383cda19494f95cfee62ca4d0ccf" member_type: NON_VOTER last_known_addr { host: "127.31.100.60" port: 37129 } attrs { promote: true } } }
I20260812 05:20:47.082994 10716 raft_consensus.cc:2932] T 00000000000000000000000000000000 P e8be870d7089424d82f5be0171ba5242 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, e8be870d7089424d82f5be0171ba5242 (127.31.100.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "070ed78c702d4333919d5d31da7b7608" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 35003 } } peers { permanent_uuid: "e8be870d7089424d82f5be0171ba5242" member_type: VOTER last_known_addr { host: "127.31.100.61" port: 34209 } attrs { promote: false } } peers { permanent_uuid: "22c6383cda19494f95cfee62ca4d0ccf" member_type: NON_VOTER last_known_addr { host: "127.31.100.60" port: 37129 } attrs { promote: true } } }
W20260812 05:20:47.083161 9718 consensus_peers.cc:487] T 00000000000000000000000000000000 P 070ed78c702d4333919d5d31da7b7608 -> Peer 22c6383cda19494f95cfee62ca4d0ccf (127.31.100.60:37129): Couldn't send request to peer 22c6383cda19494f95cfee62ca4d0ccf. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260812 05:20:47.083884 10785 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 22c6383cda19494f95cfee62ca4d0ccf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 05:20:47.084028 10785 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 22c6383cda19494f95cfee62ca4d0ccf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 05:20:47.084082 10785 tablet_replica.cc:330] T 00000000000000000000000000000000 P 22c6383cda19494f95cfee62ca4d0ccf: stopping tablet replica
I20260812 05:20:47.097676 10785 master.cc:552] Master@127.31.100.60:37129 shutdown complete.
I20260812 05:20:47.097741 10785 master_runner.cc:305] Clearing existing system tablet
I20260812 05:20:47.097872 10785 fs_manager.cc:359] Metadata directory not provided
I20260812 05:20:47.097926 10785 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 05:20:47.098642 10785 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.000s sys 0.001s
I20260812 05:20:47.099220 10946 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 05:20:47.099403 10785 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786512017613366-32144-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 05:20:47.099471 10785 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260812 05:20:47.099496 10785 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786512017613366-32144-0/minicluster-data/master-2/data,/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786512017613366-32144-0/minicluster-data/master-2/wal
uuid: "22c6383cda19494f95cfee62ca4d0ccf"
format_stamp: "Formatted at 2026-08-12 05:20:46 on dist-test-slave-42z9"
server_key: "8f263171724ab9ac250a773f65523438"
server_key_iv: "3dba32fda7cef315846a6896c4522068"
server_key_version: "encryptionkey@0"
I20260812 05:20:47.100327 10785 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 22c6383cda19494f95cfee62ca4d0ccf: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260812 05:20:47.114406 10940 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 05:20:47.114600 10940 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 05:20:47.114624 10940 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 05:20:47.115778 10940 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 05:20:47.115826 10940 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 05:20:47.115860 10940 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.100.1
I20260812 05:20:47.117585 10940 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.100.20:44403
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786512017613366-32144-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786512017613366-32144-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786512017613366-32144-0/krb5kdc/oryx_127.31.100.1.keytab
--principal=oryx/127.31.100.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.100.1:44127
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786512017613366-32144-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.100.1
--webserver_port=36363
--tserver_master_addrs=127.31.100.62:35003,127.31.100.61:34209,127.31.100.60:37129
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10940
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.100.1
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786512017613366-32144-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 04:20:34 UTC on ccycloud.quasar-phqdqp.root.comops.site
build id 4581871
I20260812 05:20:47.117807 10940 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 05:20:47.118006 10940 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 05:20:47.123638 10961 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 05:20:47.123677 10957 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 05:20:47.123692 10964 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 05:20:47.123935 10940 server_base.cc:1034] running on GCE node
I20260812 05:20:47.124135 10940 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 05:20:47.124351 10940 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 05:20:47.125504 10940 hybrid_clock.cc:648] HybridClock initialized: now 1786512047125488 us; error 28 us; skew 500 ppm
Aug 12 05:20:47 dist-test-slave-42z9 krb5kdc[9665](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512047, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.100.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 05:20:47.127912 10940 init.cc:377] Logged in from keytab as oryx/127.31.100.1@KRBTEST.COM (short username oryx)
I20260812 05:20:47.128314 10940 webserver.cc:460] Webserver started at http://127.31.100.1:36363/ using document root <none> and password file <none>
I20260812 05:20:47.128530 10940 fs_manager.cc:359] Metadata directory not provided
I20260812 05:20:47.128593 10940 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 05:20:47.130012 10940 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260812 05:20:47.130630 10971 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 05:20:47.130769 10940 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260812 05:20:47.130841 10940 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786512017613366-32144-0/minicluster-data/ts-0/data,/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786512017613366-32144-0/minicluster-data/ts-0/wal
uuid: "f54f92d5624f42a9b11d653eb7bd017c"
format_stamp: "Formatted at 2026-08-12 05:20:45 on dist-test-slave-42z9"
server_key: "acd3bb1933c8cbe38c60098ac4cb0ebe"
server_key_iv: "e5c113420c30cb2c4f92ac983df3eb1f"
server_key_version: "encryptionkey@0"
I20260812 05:20:47.131181 10940 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786512017613366-32144-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786512017613366-32144-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786512017613366-32144-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 05:20:47.149716 10940 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 05:20:47.152509 10940 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 05:20:47.152711 10940 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 05:20:47.153172 10940 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 05:20:47.153606 10940 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 05:20:47.153653 10940 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 05:20:47.153688 10940 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 05:20:47.153703 10940 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 05:20:47.170614 10940 rpc_server.cc:307] RPC server started. Bound to: 127.31.100.1:44127
I20260812 05:20:47.170795 11150 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.100.1:44127 every 8 connection(s)
I20260812 05:20:47.171093 10940 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786512017613366-32144-0/minicluster-data/ts-0/data/info.pb
I20260812 05:20:47.173393 11157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:47.172446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.60:37129 (local address 127.31.100.1:41051)
0812 05:20:47.172923 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0812 05:20:47.173060 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.100.60:37129: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":384,"thread_start_us":85,"threads_started":1}
W20260812 05:20:47.173681 11151 heartbeater.cc:644] Failed to heartbeat to 127.31.100.60:37129 (0 consecutive failures): Network error: Failed to ping master at 127.31.100.60:37129: Client connection negotiation failed: client connection to 127.31.100.60:37129: connect: Connection refused (error 111)
I20260812 05:20:47.173954 11157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:47.173843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.60:37129 (local address 127.31.100.1:60373)
0812 05:20:47.173890 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0812 05:20:47.173927 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.100.60:37129: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260812 05:20:47.174239 11157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:47.174145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.60:37129 (local address 127.31.100.1:50599)
0812 05:20:47.174186 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0812 05:20:47.174214 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.100.60:37129: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260812 05:20:47.174333 11151 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260812 05:20:47.176856 32144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0waFvT/build/release/bin/kudu as pid 10940
I20260812 05:20:47.177197 10632 sys_catalog.cc:455] T 00000000000000000000000000000000 P 070ed78c702d4333919d5d31da7b7608 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "070ed78c702d4333919d5d31da7b7608" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "070ed78c702d4333919d5d31da7b7608" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 35003 } } peers { permanent_uuid: "e8be870d7089424d82f5be0171ba5242" member_type: VOTER last_known_addr { host: "127.31.100.61" port: 34209 } attrs { promote: false } } peers { permanent_uuid: "22c6383cda19494f95cfee62ca4d0ccf" member_type: NON_VOTER last_known_addr { host: "127.31.100.60" port: 37129 } attrs { promote: true } } }
I20260812 05:20:47.177304 10632 sys_catalog.cc:458] T 00000000000000000000000000000000 P 070ed78c702d4333919d5d31da7b7608 [sys.catalog]: This master's current role is: LEADER
I20260812 05:20:47.177342 10632 sys_catalog.cc:455] T 00000000000000000000000000000000 P 070ed78c702d4333919d5d31da7b7608 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "070ed78c702d4333919d5d31da7b7608" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "070ed78c702d4333919d5d31da7b7608" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 35003 } } peers { permanent_uuid: "e8be870d7089424d82f5be0171ba5242" member_type: VOTER last_known_addr { host: "127.31.100.61" port: 34209 } attrs { promote: false } } peers { permanent_uuid: "22c6383cda19494f95cfee62ca4d0ccf" member_type: NON_VOTER last_known_addr { host: "127.31.100.60" port: 37129 } attrs { promote: true } } }
I20260812 05:20:47.177376 10632 sys_catalog.cc:458] T 00000000000000000000000000000000 P 070ed78c702d4333919d5d31da7b7608 [sys.catalog]: This master's current role is: LEADER
Aug 12 05:20:47 dist-test-slave-42z9 krb5kdc[9665](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512047, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.100.1@KRBTEST.COM for oryx/127.31.100.62@KRBTEST.COM
Aug 12 05:20:47 dist-test-slave-42z9 krb5kdc[9665](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512047, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.100.1@KRBTEST.COM for oryx/127.31.100.61@KRBTEST.COM
I20260812 05:20:47.182551 10762 sys_catalog.cc:455] T 00000000000000000000000000000000 P e8be870d7089424d82f5be0171ba5242 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "070ed78c702d4333919d5d31da7b7608" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "070ed78c702d4333919d5d31da7b7608" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 35003 } } peers { permanent_uuid: "e8be870d7089424d82f5be0171ba5242" member_type: VOTER last_known_addr { host: "127.31.100.61" port: 34209 } attrs { promote: false } } peers { permanent_uuid: "22c6383cda19494f95cfee62ca4d0ccf" member_type: NON_VOTER last_known_addr { host: "127.31.100.60" port: 37129 } attrs { promote: true } } }
I20260812 05:20:47.182641 10762 sys_catalog.cc:458] T 00000000000000000000000000000000 P e8be870d7089424d82f5be0171ba5242 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 05:20:47.182785 32144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0waFvT/build/release/bin/kudu with pid 10046
I20260812 05:20:47.191967 10893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:47.172750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.100.1:57321 (local address 127.31.100.61:34209)
0812 05:20:47.172798 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0812 05:20:47.172800 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 05:20:47.173021 (+ 221us) server_negotiation.cc:407] Connection header received
0812 05:20:47.174312 (+ 1291us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:20:47.174314 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:20:47.174326 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:20:47.174358 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:20:47.175082 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:47.175639 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:47.176986 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:47.177272 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:47.177332 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:47.180749 (+ 3417us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 05:20:47.180751 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 05:20:47.180751 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 05:20:47.180759 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0812 05:20:47.182346 (+ 1587us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:20:47.190631 (+ 8285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:20:47.190633 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:20:47.190633 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:20:47.190707 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:20:47.190976 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:20:47.190977 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:20:47.190978 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:20:47.191103 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 05:20:47.191163 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0812 05:20:47.191759 (+ 596us) server_negotiation.cc:299] Negotiation successful
0812 05:20:47.191857 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260812 05:20:47.192307 11159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:47.172628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.61:34209 (local address 127.31.100.1:57321)
0812 05:20:47.172930 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0812 05:20:47.172940 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0812 05:20:47.174217 (+ 1277us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:20:47.174430 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:20:47.174431 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:20:47.174801 (+ 370us) client_negotiation.cc:189] Negotiated authn=SASL
0812 05:20:47.174993 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:20:47.174998 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:20:47.175800 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:20:47.175801 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:20:47.176875 (+ 1074us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:20:47.176877 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:20:47.177404 (+ 527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:20:47.177405 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:20:47.177622 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:47.178210 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 05:20:47.178220 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0812 05:20:47.180621 (+ 2401us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 05:20:47.182423 (+ 1802us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 05:20:47.182424 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 05:20:47.182424 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 05:20:47.190553 (+ 8129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 05:20:47.190765 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 05:20:47.190765 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 05:20:47.190766 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 05:20:47.190905 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 05:20:47.191465 (+ 560us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 05:20:47.191473 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 05:20:47.191615 (+ 142us) client_negotiation.cc:769] Sending connection context
0812 05:20:47.191704 (+ 89us) client_negotiation.cc:240] Negotiation successful
0812 05:20:47.191879 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"mutex_wait_us":32,"thread_start_us":178,"threads_started":1}
I20260812 05:20:47.195446 10887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:47.172119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.100.1:51859 (local address 127.31.100.62:35003)
0812 05:20:47.172176 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0812 05:20:47.172178 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 05:20:47.172562 (+ 384us) server_negotiation.cc:407] Connection header received
0812 05:20:47.174160 (+ 1598us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:20:47.174161 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:20:47.174174 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:20:47.174222 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:20:47.175075 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:47.175635 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:47.177037 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:47.177324 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:47.177377 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:47.180233 (+ 2856us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 05:20:47.180235 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 05:20:47.180235 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 05:20:47.180242 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 05:20:47.182246 (+ 2004us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:20:47.182776 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:20:47.182777 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:20:47.182778 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:20:47.182846 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:20:47.194686 (+ 11840us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:20:47.194688 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:20:47.194689 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:20:47.194900 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 05:20:47.194957 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0812 05:20:47.195239 (+ 282us) server_negotiation.cc:299] Negotiation successful
0812 05:20:47.195347 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260812 05:20:47.195758 11156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:47.172122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.62:35003 (local address 127.31.100.1:51859)
0812 05:20:47.172416 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0812 05:20:47.172454 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0812 05:20:47.174028 (+ 1574us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:20:47.174304 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:20:47.174308 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:20:47.174670 (+ 362us) client_negotiation.cc:189] Negotiated authn=SASL
0812 05:20:47.174995 (+ 325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:20:47.175002 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:20:47.175819 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:20:47.175820 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:20:47.176870 (+ 1050us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:20:47.176872 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:20:47.177860 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:20:47.177860 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:20:47.177942 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:47.178361 (+ 419us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 05:20:47.178365 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0812 05:20:47.180144 (+ 1779us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 05:20:47.182341 (+ 2197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 05:20:47.182350 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 05:20:47.182352 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 05:20:47.182657 (+ 305us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 05:20:47.194462 (+ 11805us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 05:20:47.194464 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 05:20:47.194464 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 05:20:47.194559 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 05:20:47.195017 (+ 458us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 05:20:47.195018 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 05:20:47.195095 (+ 77us) client_negotiation.cc:769] Sending connection context
0812 05:20:47.195526 (+ 431us) client_negotiation.cc:240] Negotiation successful
0812 05:20:47.195624 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":236,"mutex_wait_us":33,"thread_start_us":170,"threads_started":1}
I20260812 05:20:47.206410 11154 heartbeater.cc:347] Connected to a master server at 127.31.100.61:34209
I20260812 05:20:47.206542 11154 heartbeater.cc:464] Registering TS with master...
I20260812 05:20:47.206821 11154 heartbeater.cc:511] Master 127.31.100.61:34209 requested a full tablet report, sending...
I20260812 05:20:47.207845 11152 heartbeater.cc:347] Connected to a master server at 127.31.100.62:35003
I20260812 05:20:47.207906 11152 heartbeater.cc:464] Registering TS with master...
I20260812 05:20:47.207976 11152 heartbeater.cc:511] Master 127.31.100.62:35003 requested a full tablet report, sending...
I20260812 05:20:47.208194 10699 ts_manager.cc:194] Registered new tserver with Master: f54f92d5624f42a9b11d653eb7bd017c (127.31.100.1:44127)
I20260812 05:20:47.208240 9740 ts_manager.cc:194] Re-registered known tserver with Master: f54f92d5624f42a9b11d653eb7bd017c (127.31.100.1:44127)
I20260812 05:20:47.208724 9740 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.31.100.1@KRBTEST.COM'} at 127.31.100.1:51859
I20260812 05:20:47.212313 32144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0waFvT/build/release/bin/kudu
/tmp/dist-test-task0waFvT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786512017613366-32144-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786512017613366-32144-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786512017613366-32144-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786512017613366-32144-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.100.2:36081
--local_ip_for_outbound_sockets=127.31.100.2
--tserver_master_addrs=127.31.100.62:35003
--webserver_port=39859
--webserver_interface=127.31.100.2
--keytab_file=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786512017613366-32144-0/krb5kdc/oryx_127.31.100.2.keytab
--principal=oryx/127.31.100.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.100.20:44403
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.31.100.62:35003,127.31.100.61:34209
--tserver_master_addrs=127.31.100.62:35003,127.31.100.61:34209,127.31.100.60:37129 with env {KRB5CCNAME=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786512017613366-32144-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786512017613366-32144-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786512017613366-32144-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 05:20:47.269805 10785 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 22c6383cda19494f95cfee62ca4d0ccf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260812 05:20:47.269908 10785 log.cc:1196] T 00000000000000000000000000000000 P 22c6383cda19494f95cfee62ca4d0ccf: Deleting WAL directory at /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786512017613366-32144-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260812 05:20:47.270072 10785 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 22c6383cda19494f95cfee62ca4d0ccf: Deleting consensus metadata
I20260812 05:20:47.270152 10785 master_runner.cc:315] Copying system tablet from 127.31.100.62:35003
I20260812 05:20:47.270829 10785 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 22c6383cda19494f95cfee62ca4d0ccf: tablet copy: Beginning tablet copy session from remote peer at address 127.31.100.62:35003
I20260812 05:20:47.277817 10887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:47.271225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48220 (local address 127.31.100.62:35003)
0812 05:20:47.271292 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0812 05:20:47.271294 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 05:20:47.271411 (+ 117us) server_negotiation.cc:407] Connection header received
0812 05:20:47.271487 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:20:47.271488 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:20:47.271500 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:20:47.271555 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:20:47.272063 (+ 508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:47.272597 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:47.273648 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:47.273837 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:47.273892 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:47.274980 (+ 1088us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 05:20:47.274981 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 05:20:47.274982 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 05:20:47.274988 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0812 05:20:47.276653 (+ 1665us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:20:47.277023 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:20:47.277025 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:20:47.277025 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:20:47.277092 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:20:47.277284 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:20:47.277284 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:20:47.277285 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:20:47.277421 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 05:20:47.277503 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0812 05:20:47.277645 (+ 142us) server_negotiation.cc:299] Negotiation successful
0812 05:20:47.277725 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260812 05:20:47.277828 11170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:47.271169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.62:35003 (local address 127.0.0.1:48220)
0812 05:20:47.271324 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0812 05:20:47.271335 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0812 05:20:47.271419 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:20:47.271585 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:20:47.271586 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:20:47.271893 (+ 307us) client_negotiation.cc:189] Negotiated authn=SASL
0812 05:20:47.271951 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:20:47.271954 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:20:47.272678 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:20:47.272680 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:20:47.273562 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:20:47.273566 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:20:47.273902 (+ 336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:20:47.273902 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:20:47.273957 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:47.274458 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 05:20:47.274463 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0812 05:20:47.274900 (+ 437us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 05:20:47.276742 (+ 1842us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 05:20:47.276743 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 05:20:47.276744 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 05:20:47.276948 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 05:20:47.277162 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 05:20:47.277163 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 05:20:47.277163 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 05:20:47.277228 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 05:20:47.277493 (+ 265us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 05:20:47.277494 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 05:20:47.277544 (+ 50us) client_negotiation.cc:769] Sending connection context
0812 05:20:47.277617 (+ 73us) client_negotiation.cc:240] Negotiation successful
0812 05:20:47.277728 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":98,"threads_started":1}
I20260812 05:20:47.278270 9772 tablet_copy_service.cc:140] P 070ed78c702d4333919d5d31da7b7608: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 22c6383cda19494f95cfee62ca4d0ccf ({username='oryx', principal='oryx/127.31.100.60@KRBTEST.COM'} at 127.0.0.1:48220)
I20260812 05:20:47.278334 9772 tablet_copy_service.cc:161] P 070ed78c702d4333919d5d31da7b7608: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 22c6383cda19494f95cfee62ca4d0ccf at {username='oryx', principal='oryx/127.31.100.60@KRBTEST.COM'} at 127.0.0.1:48220: session id = 22c6383cda19494f95cfee62ca4d0ccf-00000000000000000000000000000000
I20260812 05:20:47.278934 9772 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 070ed78c702d4333919d5d31da7b7608: Tablet Copy: opened 0 blocks and 1 log segments
I20260812 05:20:47.279515 10785 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260812 05:20:47.280864 10785 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 22c6383cda19494f95cfee62ca4d0ccf: tablet copy: Starting download of 0 data blocks...
I20260812 05:20:47.281003 10785 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 22c6383cda19494f95cfee62ca4d0ccf: tablet copy: Starting download of 1 WAL segments...
I20260812 05:20:47.285610 10785 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 22c6383cda19494f95cfee62ca4d0ccf: tablet copy: Tablet Copy complete. Replacing tablet superblock.
W20260812 05:20:47.302211 11160 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 05:20:47.302390 11160 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 05:20:47.302412 11160 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 05:20:47.303683 11160 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 05:20:47.303733 11160 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 05:20:47.303761 11160 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.100.2
I20260812 05:20:47.305212 11160 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.100.20:44403
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786512017613366-32144-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786512017613366-32144-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786512017613366-32144-0/krb5kdc/oryx_127.31.100.2.keytab
--principal=oryx/127.31.100.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.100.2:36081
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786512017613366-32144-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.100.2
--webserver_port=39859
--tserver_master_addrs=127.31.100.62:35003,127.31.100.61:34209,127.31.100.60:37129
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11160
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.100.2
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786512017613366-32144-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 04:20:34 UTC on ccycloud.quasar-phqdqp.root.comops.site
build id 4581871
I20260812 05:20:47.305418 11160 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 05:20:47.305688 11160 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 05:20:47.310966 11184 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 05:20:47.310963 11178 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 05:20:47.310963 11179 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 05:20:47.311105 11160 server_base.cc:1034] running on GCE node
I20260812 05:20:47.311367 11160 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 05:20:47.311585 11160 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 05:20:47.312739 11160 hybrid_clock.cc:648] HybridClock initialized: now 1786512047312724 us; error 29 us; skew 500 ppm
Aug 12 05:20:47 dist-test-slave-42z9 krb5kdc[9665](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512047, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.100.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 05:20:47.315140 11160 init.cc:377] Logged in from keytab as oryx/127.31.100.2@KRBTEST.COM (short username oryx)
I20260812 05:20:47.315608 11160 webserver.cc:460] Webserver started at http://127.31.100.2:39859/ using document root <none> and password file <none>
I20260812 05:20:47.315804 11160 fs_manager.cc:359] Metadata directory not provided
I20260812 05:20:47.315864 11160 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 05:20:47.317150 11160 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260812 05:20:47.317822 11193 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 05:20:47.317973 11160 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260812 05:20:47.318060 11160 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786512017613366-32144-0/minicluster-data/ts-1/data,/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786512017613366-32144-0/minicluster-data/ts-1/wal
uuid: "bf52456a845341ebbf4e4e7405844a2d"
format_stamp: "Formatted at 2026-08-12 05:20:45 on dist-test-slave-42z9"
server_key: "a3f014d8371e850c3a70993559019f9b"
server_key_iv: "8d0f9de869250bcb379c7d82f5d7421e"
server_key_version: "encryptionkey@0"
I20260812 05:20:47.318362 11160 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786512017613366-32144-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786512017613366-32144-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786512017613366-32144-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 05:20:47.328485 11160 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 05:20:47.330210 11160 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 05:20:47.330336 11160 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 05:20:47.330571 11160 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 05:20:47.330837 11160 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 05:20:47.330868 11160 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 05:20:47.330896 11160 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 05:20:47.330914 11160 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 05:20:47.338613 11160 rpc_server.cc:307] RPC server started. Bound to: 127.31.100.2:36081
I20260812 05:20:47.338649 11381 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.100.2:36081 every 8 connection(s)
I20260812 05:20:47.338997 11160 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786512017613366-32144-0/minicluster-data/ts-1/data/info.pb
I20260812 05:20:47.340101 11388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:47.339699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.60:37129 (local address 127.31.100.2:51263)
0812 05:20:47.339838 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0812 05:20:47.339913 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.100.60:37129: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":74,"threads_started":1}
W20260812 05:20:47.340536 11382 heartbeater.cc:644] Failed to heartbeat to 127.31.100.60:37129 (0 consecutive failures): Network error: Failed to ping master at 127.31.100.60:37129: Client connection negotiation failed: client connection to 127.31.100.60:37129: connect: Connection refused (error 111)
I20260812 05:20:47.341166 11394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:47.340719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.60:37129 (local address 127.31.100.2:37645)
0812 05:20:47.341100 (+ 381us) negotiation.cc:107] Waiting for socket to connect
0812 05:20:47.341132 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.100.60:37129: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":370}
I20260812 05:20:47.341636 11394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:47.341542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.60:37129 (local address 127.31.100.2:43849)
0812 05:20:47.341584 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0812 05:20:47.341614 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.100.60:37129: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260812 05:20:47.341751 11382 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Aug 12 05:20:47 dist-test-slave-42z9 krb5kdc[9665](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512047, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.100.2@KRBTEST.COM for oryx/127.31.100.62@KRBTEST.COM
Aug 12 05:20:47 dist-test-slave-42z9 krb5kdc[9665](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512047, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.100.2@KRBTEST.COM for oryx/127.31.100.61@KRBTEST.COM
I20260812 05:20:47.347617 32144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0waFvT/build/release/bin/kudu as pid 11160
I20260812 05:20:47.347788 32144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0waFvT/build/release/bin/kudu with pid 10264
I20260812 05:20:47.348599 10893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:47.340164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.100.2:40783 (local address 127.31.100.61:34209)
0812 05:20:47.340211 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0812 05:20:47.340213 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 05:20:47.341038 (+ 825us) server_negotiation.cc:407] Connection header received
0812 05:20:47.341080 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:20:47.341081 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:20:47.341091 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:20:47.341133 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:20:47.341900 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:47.342373 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:47.343445 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:47.343640 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:47.343691 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:47.346088 (+ 2397us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 05:20:47.346089 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 05:20:47.346089 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 05:20:47.346097 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0812 05:20:47.347340 (+ 1243us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:20:47.347691 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:20:47.347692 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:20:47.347692 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:20:47.347745 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:20:47.347931 (+ 186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:20:47.347932 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:20:47.347933 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:20:47.348046 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 05:20:47.348085 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0812 05:20:47.348430 (+ 345us) server_negotiation.cc:299] Negotiation successful
0812 05:20:47.348507 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260812 05:20:47.348613 11388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:47.339941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.62:35003 (local address 127.31.100.2:36281)
0812 05:20:47.340271 (+ 330us) negotiation.cc:107] Waiting for socket to connect
0812 05:20:47.340285 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0812 05:20:47.340833 (+ 548us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:20:47.341097 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:20:47.341101 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:20:47.341500 (+ 399us) client_negotiation.cc:189] Negotiated authn=SASL
0812 05:20:47.341771 (+ 271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:20:47.341777 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:20:47.342467 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:20:47.342468 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:20:47.343249 (+ 781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:20:47.343251 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:20:47.343582 (+ 331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:20:47.343583 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:20:47.343704 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:47.344160 (+ 456us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 05:20:47.344169 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0812 05:20:47.345631 (+ 1462us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 05:20:47.347210 (+ 1579us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 05:20:47.347216 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 05:20:47.347218 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 05:20:47.347440 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 05:20:47.347644 (+ 204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 05:20:47.347645 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 05:20:47.347645 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 05:20:47.347777 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 05:20:47.348105 (+ 328us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 05:20:47.348113 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 05:20:47.348269 (+ 156us) client_negotiation.cc:769] Sending connection context
0812 05:20:47.348373 (+ 104us) client_negotiation.cc:240] Negotiation successful
0812 05:20:47.348514 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":321,"mutex_wait_us":3,"thread_start_us":49,"threads_started":1}
I20260812 05:20:47.349326 10887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:47.340050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.100.2:36281 (local address 127.31.100.62:35003)
0812 05:20:47.340162 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0812 05:20:47.340164 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 05:20:47.340373 (+ 209us) server_negotiation.cc:407] Connection header received
0812 05:20:47.340935 (+ 562us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:20:47.340936 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:20:47.340946 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:20:47.340994 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:20:47.341868 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:47.342362 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:47.343329 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:47.343517 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:47.343570 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:47.345736 (+ 2166us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 05:20:47.345737 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 05:20:47.345738 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 05:20:47.345745 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 05:20:47.347137 (+ 1392us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:20:47.347521 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:20:47.347522 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:20:47.347523 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:20:47.347579 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:20:47.347876 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:20:47.347877 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:20:47.347878 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:20:47.348016 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 05:20:47.348076 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0812 05:20:47.349153 (+ 1077us) server_negotiation.cc:299] Negotiation successful
0812 05:20:47.349218 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260812 05:20:47.348964 11389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:47.340081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.61:34209 (local address 127.31.100.2:40783)
0812 05:20:47.340974 (+ 893us) negotiation.cc:107] Waiting for socket to connect
0812 05:20:47.340982 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0812 05:20:47.341029 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:20:47.341150 (+ 121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:20:47.341151 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:20:47.341513 (+ 362us) client_negotiation.cc:189] Negotiated authn=SASL
0812 05:20:47.341777 (+ 264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:20:47.341780 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:20:47.342535 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:20:47.342536 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:20:47.343370 (+ 834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:20:47.343372 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:20:47.343696 (+ 324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:20:47.343697 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:20:47.343802 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:47.344297 (+ 495us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 05:20:47.344301 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0812 05:20:47.346012 (+ 1711us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 05:20:47.347413 (+ 1401us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 05:20:47.347414 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 05:20:47.347415 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 05:20:47.347610 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 05:20:47.347806 (+ 196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 05:20:47.347807 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 05:20:47.347807 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 05:20:47.347863 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 05:20:47.348129 (+ 266us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 05:20:47.348130 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 05:20:47.348269 (+ 139us) client_negotiation.cc:769] Sending connection context
0812 05:20:47.348725 (+ 456us) client_negotiation.cc:240] Negotiation successful
0812 05:20:47.348826 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":880,"thread_start_us":57,"threads_started":1}
I20260812 05:20:47.371986 11385 heartbeater.cc:347] Connected to a master server at 127.31.100.61:34209
I20260812 05:20:47.371986 11384 heartbeater.cc:347] Connected to a master server at 127.31.100.62:35003
I20260812 05:20:47.372126 11385 heartbeater.cc:464] Registering TS with master...
I20260812 05:20:47.372126 11384 heartbeater.cc:464] Registering TS with master...
I20260812 05:20:47.372393 11384 heartbeater.cc:511] Master 127.31.100.62:35003 requested a full tablet report, sending...
I20260812 05:20:47.372392 11385 heartbeater.cc:511] Master 127.31.100.61:34209 requested a full tablet report, sending...
I20260812 05:20:47.372864 9740 ts_manager.cc:194] Re-registered known tserver with Master: bf52456a845341ebbf4e4e7405844a2d (127.31.100.2:36081)
I20260812 05:20:47.373383 9740 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.31.100.2@KRBTEST.COM'} at 127.31.100.2:36281
I20260812 05:20:47.373672 10699 ts_manager.cc:194] Registered new tserver with Master: bf52456a845341ebbf4e4e7405844a2d (127.31.100.2:36081)
I20260812 05:20:47.376076 32144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0waFvT/build/release/bin/kudu
/tmp/dist-test-task0waFvT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786512017613366-32144-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786512017613366-32144-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786512017613366-32144-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786512017613366-32144-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.100.3:39085
--local_ip_for_outbound_sockets=127.31.100.3
--tserver_master_addrs=127.31.100.62:35003
--webserver_port=39337
--webserver_interface=127.31.100.3
--keytab_file=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786512017613366-32144-0/krb5kdc/oryx_127.31.100.3.keytab
--principal=oryx/127.31.100.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.100.20:44403
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.31.100.62:35003,127.31.100.61:34209
--tserver_master_addrs=127.31.100.62:35003,127.31.100.61:34209,127.31.100.60:37129 with env {KRB5CCNAME=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786512017613366-32144-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786512017613366-32144-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786512017613366-32144-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 05:20:47.378510 9772 tablet_copy_service.cc:342] P 070ed78c702d4333919d5d31da7b7608: Request end of tablet copy session 22c6383cda19494f95cfee62ca4d0ccf-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.31.100.60@KRBTEST.COM'} at 127.0.0.1:48220
I20260812 05:20:47.378572 9772 tablet_copy_service.cc:433] P 070ed78c702d4333919d5d31da7b7608: ending tablet copy session 22c6383cda19494f95cfee62ca4d0ccf-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 22c6383cda19494f95cfee62ca4d0ccf
I20260812 05:20:47.381033 10785 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 05:20:47.382283 11396 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 05:20:47.382258 11399 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 05:20:47.382400 11397 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 05:20:47.382483 10785 server_base.cc:1034] running on GCE node
I20260812 05:20:47.382619 10785 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 05:20:47.382750 10785 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 05:20:47.383895 10785 hybrid_clock.cc:648] HybridClock initialized: now 1786512047383886 us; error 25 us; skew 500 ppm
Aug 12 05:20:47 dist-test-slave-42z9 krb5kdc[9665](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512047, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.100.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 05:20:47.385130 10785 init.cc:377] Logged in from keytab as oryx/127.31.100.60@KRBTEST.COM (short username oryx)
I20260812 05:20:47.385365 10785 webserver.cc:460] Webserver started at http://127.31.100.60:38121/ using document root <none> and password file <none>
I20260812 05:20:47.385566 10785 fs_manager.cc:359] Metadata directory not provided
I20260812 05:20:47.385608 10785 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 05:20:47.386189 10785 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.000s sys 0.001s
I20260812 05:20:47.386716 11415 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 05:20:47.386826 10785 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260812 05:20:47.386868 10785 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786512017613366-32144-0/minicluster-data/master-2/data,/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786512017613366-32144-0/minicluster-data/master-2/wal
uuid: "22c6383cda19494f95cfee62ca4d0ccf"
format_stamp: "Formatted at 2026-08-12 05:20:46 on dist-test-slave-42z9"
server_key: "8f263171724ab9ac250a773f65523438"
server_key_iv: "3dba32fda7cef315846a6896c4522068"
server_key_version: "encryptionkey@0"
I20260812 05:20:47.386924 10785 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786512017613366-32144-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786512017613366-32144-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786512017613366-32144-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 05:20:47.401680 10785 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 05:20:47.403131 10785 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 05:20:47.405931 10785 rpc_server.cc:307] RPC server started. Bound to: 127.31.100.60:37129
I20260812 05:20:47.406343 10785 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786512017613366-32144-0/minicluster-data/master-2/data/info.pb
I20260812 05:20:47.406956 11496 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.100.60:37129 every 8 connection(s)
I20260812 05:20:47.407604 11499 sys_catalog.cc:263] Verifying existing consensus state
I20260812 05:20:47.408179 11499 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 22c6383cda19494f95cfee62ca4d0ccf: Bootstrap starting.
I20260812 05:20:47.411046 11499 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 22c6383cda19494f95cfee62ca4d0ccf: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 05:20:47.411244 11499 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 22c6383cda19494f95cfee62ca4d0ccf: Bootstrap complete.
I20260812 05:20:47.411418 11499 raft_consensus.cc:348] T 00000000000000000000000000000000 P 22c6383cda19494f95cfee62ca4d0ccf [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "070ed78c702d4333919d5d31da7b7608" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 35003 } } peers { permanent_uuid: "e8be870d7089424d82f5be0171ba5242" member_type: VOTER last_known_addr { host: "127.31.100.61" port: 34209 } attrs { promote: false } } peers { permanent_uuid: "22c6383cda19494f95cfee62ca4d0ccf" member_type: NON_VOTER last_known_addr { host: "127.31.100.60" port: 37129 } attrs { promote: true } }
I20260812 05:20:47.411506 11499 raft_consensus.cc:729] T 00000000000000000000000000000000 P 22c6383cda19494f95cfee62ca4d0ccf [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 22c6383cda19494f95cfee62ca4d0ccf, State: Initialized, Role: LEARNER
I20260812 05:20:47.411549 11499 consensus_queue.cc:260] T 00000000000000000000000000000000 P 22c6383cda19494f95cfee62ca4d0ccf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "070ed78c702d4333919d5d31da7b7608" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 35003 } } peers { permanent_uuid: "e8be870d7089424d82f5be0171ba5242" member_type: VOTER last_known_addr { host: "127.31.100.61" port: 34209 } attrs { promote: false } } peers { permanent_uuid: "22c6383cda19494f95cfee62ca4d0ccf" member_type: NON_VOTER last_known_addr { host: "127.31.100.60" port: 37129 } attrs { promote: true } }
I20260812 05:20:47.411718 11499 sys_catalog.cc:564] T 00000000000000000000000000000000 P 22c6383cda19494f95cfee62ca4d0ccf [sys.catalog]: configured and running, proceeding with master startup.
I20260812 05:20:47.411716 11502 sys_catalog.cc:455] T 00000000000000000000000000000000 P 22c6383cda19494f95cfee62ca4d0ccf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "070ed78c702d4333919d5d31da7b7608" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 35003 } } peers { permanent_uuid: "e8be870d7089424d82f5be0171ba5242" member_type: VOTER last_known_addr { host: "127.31.100.61" port: 34209 } attrs { promote: false } } peers { permanent_uuid: "22c6383cda19494f95cfee62ca4d0ccf" member_type: NON_VOTER last_known_addr { host: "127.31.100.60" port: 37129 } attrs { promote: true } } }
I20260812 05:20:47.411821 11502 sys_catalog.cc:458] T 00000000000000000000000000000000 P 22c6383cda19494f95cfee62ca4d0ccf [sys.catalog]: This master's current role is: LEARNER
I20260812 05:20:47.413159 11517 catalog_manager.cc:1259] Loaded cluster ID: 1c12969b4cd344eab17b8b9fb1e66566
I20260812 05:20:47.413208 11517 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 22c6383cda19494f95cfee62ca4d0ccf: loading cluster ID for follower catalog manager: success
I20260812 05:20:47.414283 11517 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 22c6383cda19494f95cfee62ca4d0ccf: acquiring CA information for follower catalog manager: success
I20260812 05:20:47.414510 11517 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 22c6383cda19494f95cfee62ca4d0ccf: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260812 05:20:47.463212 11395 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 05:20:47.463403 11395 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 05:20:47.463434 11395 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 05:20:47.464540 11395 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 05:20:47.464581 11395 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 05:20:47.464613 11395 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.100.3
I20260812 05:20:47.466084 11395 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.100.20:44403
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786512017613366-32144-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786512017613366-32144-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786512017613366-32144-0/krb5kdc/oryx_127.31.100.3.keytab
--principal=oryx/127.31.100.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.100.3:39085
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786512017613366-32144-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.100.3
--webserver_port=39337
--tserver_master_addrs=127.31.100.62:35003,127.31.100.61:34209,127.31.100.60:37129
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11395
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.100.3
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786512017613366-32144-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 04:20:34 UTC on ccycloud.quasar-phqdqp.root.comops.site
build id 4581871
I20260812 05:20:47.466300 11395 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 05:20:47.466504 11395 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 05:20:47.471280 11528 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 05:20:47.471318 11533 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 05:20:47.471455 11395 server_base.cc:1034] running on GCE node
W20260812 05:20:47.471289 11529 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 05:20:47.471757 11395 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 05:20:47.471994 11395 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 05:20:47.473125 11395 hybrid_clock.cc:648] HybridClock initialized: now 1786512047473114 us; error 26 us; skew 500 ppm
Aug 12 05:20:47 dist-test-slave-42z9 krb5kdc[9665](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512047, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.100.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 05:20:47.475430 11395 init.cc:377] Logged in from keytab as oryx/127.31.100.3@KRBTEST.COM (short username oryx)
I20260812 05:20:47.475806 11395 webserver.cc:460] Webserver started at http://127.31.100.3:39337/ using document root <none> and password file <none>
I20260812 05:20:47.476001 11395 fs_manager.cc:359] Metadata directory not provided
I20260812 05:20:47.476065 11395 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 05:20:47.477293 11395 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260812 05:20:47.477833 11544 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 05:20:47.477989 11395 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260812 05:20:47.478070 11395 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786512017613366-32144-0/minicluster-data/ts-2/data,/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786512017613366-32144-0/minicluster-data/ts-2/wal
uuid: "167b45749e0d4747866e5be16306875f"
format_stamp: "Formatted at 2026-08-12 05:20:45 on dist-test-slave-42z9"
server_key: "f7161c65bf753427bc7aa1056c49ec23"
server_key_iv: "e519440e368b175644e8c7a995ac3470"
server_key_version: "encryptionkey@0"
I20260812 05:20:47.478360 11395 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786512017613366-32144-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786512017613366-32144-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786512017613366-32144-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 05:20:47.495473 11395 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 05:20:47.497267 11395 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 05:20:47.497396 11395 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 05:20:47.497619 11395 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 05:20:47.497895 11395 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 05:20:47.497929 11395 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 05:20:47.497967 11395 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 05:20:47.497991 11395 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 05:20:47.503649 11395 rpc_server.cc:307] RPC server started. Bound to: 127.31.100.3:39085
I20260812 05:20:47.503702 11749 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.100.3:39085 every 8 connection(s)
I20260812 05:20:47.504021 11395 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786512017613366-32144-0/minicluster-data/ts-2/data/info.pb
Aug 12 05:20:47 dist-test-slave-42z9 krb5kdc[9665](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512047, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.100.3@KRBTEST.COM for oryx/127.31.100.62@KRBTEST.COM
Aug 12 05:20:47 dist-test-slave-42z9 krb5kdc[9665](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512047, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.100.3@KRBTEST.COM for oryx/127.31.100.61@KRBTEST.COM
Aug 12 05:20:47 dist-test-slave-42z9 krb5kdc[9665](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512047, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.100.3@KRBTEST.COM for oryx/127.31.100.60@KRBTEST.COM
I20260812 05:20:47.512209 32144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0waFvT/build/release/bin/kudu as pid 11395
I20260812 05:20:47.515169 10887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:47.504660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.100.3:57671 (local address 127.31.100.62:35003)
0812 05:20:47.504713 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0812 05:20:47.504716 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 05:20:47.504937 (+ 221us) server_negotiation.cc:407] Connection header received
0812 05:20:47.505606 (+ 669us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:20:47.505607 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:20:47.505620 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:20:47.505669 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:20:47.506586 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:47.507353 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:47.508836 (+ 1483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:47.509125 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:47.509180 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:47.511590 (+ 2410us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 05:20:47.511592 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 05:20:47.511593 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 05:20:47.511601 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0812 05:20:47.513515 (+ 1914us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:20:47.514079 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:20:47.514080 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:20:47.514081 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:20:47.514140 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:20:47.514419 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:20:47.514420 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:20:47.514420 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:20:47.514566 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 05:20:47.514607 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0812 05:20:47.514975 (+ 368us) server_negotiation.cc:299] Negotiation successful
0812 05:20:47.515059 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260812 05:20:47.515362 10893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:47.505295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.100.3:58131 (local address 127.31.100.61:34209)
0812 05:20:47.505358 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0812 05:20:47.505360 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 05:20:47.505367 (+ 7us) server_negotiation.cc:407] Connection header received
0812 05:20:47.505719 (+ 352us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:20:47.505721 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:20:47.505729 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:20:47.505770 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:20:47.506576 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:47.507354 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:47.508801 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:47.509092 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:47.509156 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:47.511954 (+ 2798us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 05:20:47.511955 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 05:20:47.511956 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 05:20:47.511963 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 05:20:47.514228 (+ 2265us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:20:47.514557 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:20:47.514558 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:20:47.514558 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:20:47.514615 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:20:47.514839 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:20:47.514840 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:20:47.514840 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:20:47.514979 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 05:20:47.515018 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0812 05:20:47.515203 (+ 185us) server_negotiation.cc:299] Negotiation successful
0812 05:20:47.515281 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260812 05:20:47.515789 11761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:47.505003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.100.3:56115 (local address 127.31.100.60:37129)
0812 05:20:47.505119 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0812 05:20:47.505121 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 05:20:47.505131 (+ 10us) server_negotiation.cc:407] Connection header received
0812 05:20:47.505827 (+ 696us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:20:47.505828 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:20:47.505842 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:20:47.505910 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:20:47.506604 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:47.507596 (+ 992us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:47.508827 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:47.509058 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:47.509090 (+ 32us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:47.512445 (+ 3355us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 05:20:47.512447 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 05:20:47.512447 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 05:20:47.512456 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0812 05:20:47.514574 (+ 2118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:20:47.514975 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:20:47.514976 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:20:47.514977 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:20:47.515038 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:20:47.515241 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:20:47.515241 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:20:47.515242 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:20:47.515372 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 05:20:47.515412 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0812 05:20:47.515608 (+ 196us) server_negotiation.cc:299] Negotiation successful
0812 05:20:47.515690 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86,"thread_start_us":53,"threads_started":1}
I20260812 05:20:47.515892 11762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:47.505095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.61:34209 (local address 127.31.100.3:58131)
0812 05:20:47.505209 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0812 05:20:47.505220 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0812 05:20:47.505507 (+ 287us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:20:47.505786 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:20:47.505787 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:20:47.506189 (+ 402us) client_negotiation.cc:189] Negotiated authn=SASL
0812 05:20:47.506472 (+ 283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:20:47.506479 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:20:47.507652 (+ 1173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:20:47.507653 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:20:47.508721 (+ 1068us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:20:47.508724 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:20:47.509172 (+ 448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:20:47.509174 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:20:47.509321 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:47.509913 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 05:20:47.509916 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0812 05:20:47.511884 (+ 1968us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 05:20:47.514301 (+ 2417us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 05:20:47.514301 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 05:20:47.514302 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 05:20:47.514483 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 05:20:47.514681 (+ 198us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 05:20:47.514682 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 05:20:47.514682 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 05:20:47.514770 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 05:20:47.515063 (+ 293us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 05:20:47.515064 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 05:20:47.515101 (+ 37us) client_negotiation.cc:769] Sending connection context
0812 05:20:47.515487 (+ 386us) client_negotiation.cc:240] Negotiation successful
0812 05:20:47.515661 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":98,"mutex_wait_us":58,"thread_start_us":62,"threads_started":1}
I20260812 05:20:47.516252 11759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:47.504661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.62:35003 (local address 127.31.100.3:57671)
0812 05:20:47.504799 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0812 05:20:47.504833 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0812 05:20:47.505501 (+ 668us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:20:47.505740 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:20:47.505744 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:20:47.506276 (+ 532us) client_negotiation.cc:189] Negotiated authn=SASL
0812 05:20:47.506471 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:20:47.506477 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:20:47.507487 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:20:47.507488 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:20:47.508727 (+ 1239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:20:47.508730 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:20:47.509228 (+ 498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:20:47.509229 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:20:47.509317 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:47.509980 (+ 663us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 05:20:47.509984 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0812 05:20:47.511503 (+ 1519us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 05:20:47.513706 (+ 2203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 05:20:47.513713 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 05:20:47.513716 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 05:20:47.513984 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 05:20:47.514209 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 05:20:47.514210 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 05:20:47.514210 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 05:20:47.514331 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 05:20:47.514653 (+ 322us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 05:20:47.514660 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 05:20:47.514811 (+ 151us) client_negotiation.cc:769] Sending connection context
0812 05:20:47.515822 (+ 1011us) client_negotiation.cc:240] Negotiation successful
0812 05:20:47.516156 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"mutex_wait_us":12,"thread_start_us":77,"threads_started":1}
I20260812 05:20:47.516355 11753 heartbeater.cc:347] Connected to a master server at 127.31.100.61:34209
I20260812 05:20:47.516575 11753 heartbeater.cc:464] Registering TS with master...
I20260812 05:20:47.516757 11753 heartbeater.cc:511] Master 127.31.100.61:34209 requested a full tablet report, sending...
I20260812 05:20:47.517187 10699 ts_manager.cc:194] Registered new tserver with Master: 167b45749e0d4747866e5be16306875f (127.31.100.3:39085)
I20260812 05:20:47.517625 11751 heartbeater.cc:347] Connected to a master server at 127.31.100.62:35003
I20260812 05:20:47.517675 11751 heartbeater.cc:464] Registering TS with master...
I20260812 05:20:47.517732 11751 heartbeater.cc:511] Master 127.31.100.62:35003 requested a full tablet report, sending...
I20260812 05:20:47.517938 11760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:47.504880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.60:37129 (local address 127.31.100.3:56115)
0812 05:20:47.505037 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0812 05:20:47.505047 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0812 05:20:47.505505 (+ 458us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:20:47.505936 (+ 431us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:20:47.505937 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:20:47.506190 (+ 253us) client_negotiation.cc:189] Negotiated authn=SASL
0812 05:20:47.506471 (+ 281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:20:47.506477 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:20:47.507678 (+ 1201us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:20:47.507679 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:20:47.508727 (+ 1048us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:20:47.508730 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:20:47.509174 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:20:47.509175 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:20:47.509321 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:47.509899 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 05:20:47.509909 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0812 05:20:47.512370 (+ 2461us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 05:20:47.514651 (+ 2281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 05:20:47.514652 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 05:20:47.514652 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 05:20:47.514878 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 05:20:47.515111 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 05:20:47.515112 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 05:20:47.515112 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 05:20:47.515170 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 05:20:47.515459 (+ 289us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 05:20:47.515460 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 05:20:47.515501 (+ 41us) client_negotiation.cc:769] Sending connection context
0812 05:20:47.515940 (+ 439us) client_negotiation.cc:240] Negotiation successful
0812 05:20:47.517835 (+ 1895us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"mutex_wait_us":86,"thread_start_us":109,"threads_started":1}
I20260812 05:20:47.517944 9740 ts_manager.cc:194] Re-registered known tserver with Master: 167b45749e0d4747866e5be16306875f (127.31.100.3:39085)
I20260812 05:20:47.518249 11750 heartbeater.cc:347] Connected to a master server at 127.31.100.60:37129
I20260812 05:20:47.518371 9740 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.31.100.3@KRBTEST.COM'} at 127.31.100.3:57671
I20260812 05:20:47.518289 11750 heartbeater.cc:464] Registering TS with master...
I20260812 05:20:47.518771 11750 heartbeater.cc:511] Master 127.31.100.60:37129 requested a full tablet report, sending...
I20260812 05:20:47.519178 11441 ts_manager.cc:194] Registered new tserver with Master: 167b45749e0d4747866e5be16306875f (127.31.100.3:39085)
I20260812 05:20:47.521924 10887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:47.513492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48228 (local address 127.31.100.62:35003)
0812 05:20:47.515243 (+ 1751us) server_negotiation.cc:206] Beginning negotiation
0812 05:20:47.515245 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 05:20:47.515251 (+ 6us) server_negotiation.cc:407] Connection header received
0812 05:20:47.515269 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:20:47.515270 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:20:47.515278 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:20:47.515325 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:20:47.516168 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:47.516800 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:47.517751 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:47.518011 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:47.518051 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:47.519491 (+ 1440us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 05:20:47.519493 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 05:20:47.519493 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 05:20:47.519500 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 05:20:47.520816 (+ 1316us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:20:47.521166 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:20:47.521167 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:20:47.521167 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:20:47.521220 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:20:47.521414 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:20:47.521414 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:20:47.521415 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:20:47.521538 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 05:20:47.521624 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0812 05:20:47.521768 (+ 144us) server_negotiation.cc:299] Negotiation successful
0812 05:20:47.521842 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1737,"thread_start_us":66,"threads_started":1}
I20260812 05:20:47.522467 11773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:47.513717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49876 (local address 127.31.100.61:34209)
0812 05:20:47.514468 (+ 751us) server_negotiation.cc:206] Beginning negotiation
0812 05:20:47.514470 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 05:20:47.514610 (+ 140us) server_negotiation.cc:407] Connection header received
0812 05:20:47.514632 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:20:47.514633 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:20:47.514645 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:20:47.514697 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:20:47.516180 (+ 1483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:47.516870 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:47.517821 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:47.518130 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:47.518187 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:47.519652 (+ 1465us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 05:20:47.519654 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 05:20:47.519654 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 05:20:47.519661 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 05:20:47.521192 (+ 1531us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:20:47.521574 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:20:47.521575 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:20:47.521576 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:20:47.521630 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:20:47.521813 (+ 183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:20:47.521814 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:20:47.521814 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:20:47.521940 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 05:20:47.522132 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0812 05:20:47.522276 (+ 144us) server_negotiation.cc:299] Negotiation successful
0812 05:20:47.522350 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":722,"thread_start_us":60,"threads_started":1}
I20260812 05:20:47.523558 9740 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48228:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 05:20:47.525708 9740 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 05:20:47.526048 11776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:47.514037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55760 (local address 127.31.100.60:37129)
0812 05:20:47.514268 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0812 05:20:47.514270 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 05:20:47.518551 (+ 4281us) server_negotiation.cc:407] Connection header received
0812 05:20:47.519353 (+ 802us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:20:47.519354 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:20:47.519383 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:20:47.519429 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:20:47.520198 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:47.520896 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:47.521931 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:47.522201 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:47.522239 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:47.523576 (+ 1337us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 05:20:47.523577 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 05:20:47.523577 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 05:20:47.523583 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0812 05:20:47.524989 (+ 1406us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:20:47.525279 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:20:47.525279 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:20:47.525280 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:20:47.525338 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:20:47.525528 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:20:47.525528 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:20:47.525529 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:20:47.525652 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 05:20:47.525738 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0812 05:20:47.525893 (+ 155us) server_negotiation.cc:299] Negotiation successful
0812 05:20:47.525972 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":205,"thread_start_us":63,"threads_started":1}
I20260812 05:20:47.533223 11788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:47.529245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.1:44127 (local address 127.0.0.1:60818)
0812 05:20:47.529365 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0812 05:20:47.529376 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0812 05:20:47.529423 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:20:47.529980 (+ 557us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:20:47.529981 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:20:47.529983 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 05:20:47.530019 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:20:47.530021 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:20:47.531237 (+ 1216us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:20:47.531238 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:20:47.532148 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:20:47.532151 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:20:47.533005 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:20:47.533006 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:20:47.533051 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:47.533068 (+ 17us) client_negotiation.cc:769] Sending connection context
0812 05:20:47.533114 (+ 46us) client_negotiation.cc:240] Negotiation successful
0812 05:20:47.533154 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":83,"threads_started":1}
I20260812 05:20:47.533263 11795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:47.529486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60818 (local address 127.31.100.1:44127)
0812 05:20:47.529803 (+ 317us) server_negotiation.cc:206] Beginning negotiation
0812 05:20:47.529807 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 05:20:47.529818 (+ 11us) server_negotiation.cc:407] Connection header received
0812 05:20:47.529857 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:20:47.529859 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:20:47.529876 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:20:47.529951 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 05:20:47.530073 (+ 122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:47.531142 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:47.532227 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:47.532917 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:47.532986 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:47.533046 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0812 05:20:47.533128 (+ 82us) server_negotiation.cc:299] Negotiation successful
0812 05:20:47.533167 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":277,"thread_start_us":83,"threads_started":1}
I20260812 05:20:47.533332 11790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:47.529311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.2:36081 (local address 127.0.0.1:41760)
0812 05:20:47.529780 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0812 05:20:47.529788 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0812 05:20:47.529820 (+ 32us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:20:47.530296 (+ 476us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:20:47.530296 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:20:47.530298 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 05:20:47.530326 (+ 28us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:20:47.530327 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:20:47.531155 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:20:47.531156 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:20:47.532263 (+ 1107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:20:47.532265 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:20:47.533112 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:20:47.533113 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:20:47.533175 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:47.533179 (+ 4us) client_negotiation.cc:769] Sending connection context
0812 05:20:47.533224 (+ 45us) client_negotiation.cc:240] Negotiation successful
0812 05:20:47.533276 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":457,"thread_start_us":72,"threads_started":1}
I20260812 05:20:47.533625 11798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:47.529558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41760 (local address 127.31.100.2:36081)
0812 05:20:47.530155 (+ 597us) server_negotiation.cc:206] Beginning negotiation
0812 05:20:47.530159 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 05:20:47.530167 (+ 8us) server_negotiation.cc:407] Connection header received
0812 05:20:47.530200 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:20:47.530202 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:20:47.530216 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:20:47.530288 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 05:20:47.530420 (+ 132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:47.531078 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:47.532350 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:47.533035 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:47.533099 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:47.533161 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0812 05:20:47.533498 (+ 337us) server_negotiation.cc:299] Negotiation successful
0812 05:20:47.533542 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":560,"thread_start_us":71,"threads_started":1}
I20260812 05:20:47.534292 11043 tablet_service.cc:1467] Processing CreateTablet for tablet 4b9630239c344b0ab7ec8bc4ff034a62 (DEFAULT_TABLE table=test-table [id=d43079d434a5441eb12631a4cd24d718]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 05:20:47.534732 11043 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4b9630239c344b0ab7ec8bc4ff034a62. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 05:20:47.534755 11789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:47.529288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.3:39085 (local address 127.0.0.1:44782)
0812 05:20:47.529658 (+ 370us) negotiation.cc:107] Waiting for socket to connect
0812 05:20:47.529668 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0812 05:20:47.529709 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:20:47.530052 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:20:47.530053 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:20:47.530055 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 05:20:47.530084 (+ 29us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:20:47.530085 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:20:47.531413 (+ 1328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:20:47.531414 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:20:47.532451 (+ 1037us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:20:47.532453 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:20:47.534553 (+ 2100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:20:47.534554 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:20:47.534622 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:47.534630 (+ 8us) client_negotiation.cc:769] Sending connection context
0812 05:20:47.534671 (+ 41us) client_negotiation.cc:240] Negotiation successful
0812 05:20:47.534700 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":351,"thread_start_us":95,"threads_started":1}
I20260812 05:20:47.534806 11797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:47.529540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44782 (local address 127.31.100.3:39085)
0812 05:20:47.529913 (+ 373us) server_negotiation.cc:206] Beginning negotiation
0812 05:20:47.529917 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 05:20:47.529924 (+ 7us) server_negotiation.cc:407] Connection header received
0812 05:20:47.529963 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:20:47.529965 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:20:47.529977 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:20:47.530045 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 05:20:47.530362 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:47.531337 (+ 975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:47.533194 (+ 1857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:47.534479 (+ 1285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:47.534549 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:47.534604 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0812 05:20:47.534685 (+ 81us) server_negotiation.cc:299] Negotiation successful
0812 05:20:47.534722 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":343,"thread_start_us":87,"threads_started":1}
I20260812 05:20:47.535409 11630 tablet_service.cc:1467] Processing CreateTablet for tablet 4b9630239c344b0ab7ec8bc4ff034a62 (DEFAULT_TABLE table=test-table [id=d43079d434a5441eb12631a4cd24d718]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 05:20:47.535712 11630 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4b9630239c344b0ab7ec8bc4ff034a62. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 05:20:47.538074 11801 tablet_bootstrap.cc:492] T 4b9630239c344b0ab7ec8bc4ff034a62 P f54f92d5624f42a9b11d653eb7bd017c: Bootstrap starting.
I20260812 05:20:47.538425 11803 tablet_bootstrap.cc:492] T 4b9630239c344b0ab7ec8bc4ff034a62 P 167b45749e0d4747866e5be16306875f: Bootstrap starting.
I20260812 05:20:47.538483 11275 tablet_service.cc:1467] Processing CreateTablet for tablet 4b9630239c344b0ab7ec8bc4ff034a62 (DEFAULT_TABLE table=test-table [id=d43079d434a5441eb12631a4cd24d718]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 05:20:47.538848 11275 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4b9630239c344b0ab7ec8bc4ff034a62. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 05:20:47.538864 11801 tablet_bootstrap.cc:654] T 4b9630239c344b0ab7ec8bc4ff034a62 P f54f92d5624f42a9b11d653eb7bd017c: Neither blocks nor log segments found. Creating new log.
I20260812 05:20:47.539036 11803 tablet_bootstrap.cc:654] T 4b9630239c344b0ab7ec8bc4ff034a62 P 167b45749e0d4747866e5be16306875f: Neither blocks nor log segments found. Creating new log.
I20260812 05:20:47.539291 11803 log.cc:824] T 4b9630239c344b0ab7ec8bc4ff034a62 P 167b45749e0d4747866e5be16306875f: Log is configured to *not* fsync() on all Append() calls
I20260812 05:20:47.539423 11801 log.cc:824] T 4b9630239c344b0ab7ec8bc4ff034a62 P f54f92d5624f42a9b11d653eb7bd017c: Log is configured to *not* fsync() on all Append() calls
I20260812 05:20:47.540411 11801 tablet_bootstrap.cc:492] T 4b9630239c344b0ab7ec8bc4ff034a62 P f54f92d5624f42a9b11d653eb7bd017c: No bootstrap required, opened a new log
I20260812 05:20:47.540498 11801 ts_tablet_manager.cc:1397] T 4b9630239c344b0ab7ec8bc4ff034a62 P f54f92d5624f42a9b11d653eb7bd017c: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260812 05:20:47.543437 11808 tablet_bootstrap.cc:492] T 4b9630239c344b0ab7ec8bc4ff034a62 P bf52456a845341ebbf4e4e7405844a2d: Bootstrap starting.
I20260812 05:20:47.543777 11801 raft_consensus.cc:348] T 4b9630239c344b0ab7ec8bc4ff034a62 P f54f92d5624f42a9b11d653eb7bd017c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54f92d5624f42a9b11d653eb7bd017c" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 44127 } } peers { permanent_uuid: "167b45749e0d4747866e5be16306875f" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 39085 } } peers { permanent_uuid: "bf52456a845341ebbf4e4e7405844a2d" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 36081 } }
I20260812 05:20:47.543915 11801 raft_consensus.cc:374] T 4b9630239c344b0ab7ec8bc4ff034a62 P f54f92d5624f42a9b11d653eb7bd017c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 05:20:47.543937 11801 raft_consensus.cc:729] T 4b9630239c344b0ab7ec8bc4ff034a62 P f54f92d5624f42a9b11d653eb7bd017c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f54f92d5624f42a9b11d653eb7bd017c, State: Initialized, Role: FOLLOWER
I20260812 05:20:47.544062 11808 tablet_bootstrap.cc:654] T 4b9630239c344b0ab7ec8bc4ff034a62 P bf52456a845341ebbf4e4e7405844a2d: Neither blocks nor log segments found. Creating new log.
I20260812 05:20:47.544071 11801 consensus_queue.cc:260] T 4b9630239c344b0ab7ec8bc4ff034a62 P f54f92d5624f42a9b11d653eb7bd017c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54f92d5624f42a9b11d653eb7bd017c" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 44127 } } peers { permanent_uuid: "167b45749e0d4747866e5be16306875f" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 39085 } } peers { permanent_uuid: "bf52456a845341ebbf4e4e7405844a2d" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 36081 } }
I20260812 05:20:47.544202 11803 tablet_bootstrap.cc:492] T 4b9630239c344b0ab7ec8bc4ff034a62 P 167b45749e0d4747866e5be16306875f: No bootstrap required, opened a new log
I20260812 05:20:47.544296 11803 ts_tablet_manager.cc:1397] T 4b9630239c344b0ab7ec8bc4ff034a62 P 167b45749e0d4747866e5be16306875f: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20260812 05:20:47.544385 11152 heartbeater.cc:503] Master 127.31.100.62:35003 was elected leader, sending a full tablet report...
I20260812 05:20:47.544723 11808 log.cc:824] T 4b9630239c344b0ab7ec8bc4ff034a62 P bf52456a845341ebbf4e4e7405844a2d: Log is configured to *not* fsync() on all Append() calls
I20260812 05:20:47.545233 11801 ts_tablet_manager.cc:1428] T 4b9630239c344b0ab7ec8bc4ff034a62 P f54f92d5624f42a9b11d653eb7bd017c: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260812 05:20:47.545905 11808 tablet_bootstrap.cc:492] T 4b9630239c344b0ab7ec8bc4ff034a62 P bf52456a845341ebbf4e4e7405844a2d: No bootstrap required, opened a new log
I20260812 05:20:47.545980 11808 ts_tablet_manager.cc:1397] T 4b9630239c344b0ab7ec8bc4ff034a62 P bf52456a845341ebbf4e4e7405844a2d: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260812 05:20:47.546983 11803 raft_consensus.cc:348] T 4b9630239c344b0ab7ec8bc4ff034a62 P 167b45749e0d4747866e5be16306875f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54f92d5624f42a9b11d653eb7bd017c" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 44127 } } peers { permanent_uuid: "167b45749e0d4747866e5be16306875f" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 39085 } } peers { permanent_uuid: "bf52456a845341ebbf4e4e7405844a2d" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 36081 } }
I20260812 05:20:47.547142 11803 raft_consensus.cc:374] T 4b9630239c344b0ab7ec8bc4ff034a62 P 167b45749e0d4747866e5be16306875f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 05:20:47.547185 11803 raft_consensus.cc:729] T 4b9630239c344b0ab7ec8bc4ff034a62 P 167b45749e0d4747866e5be16306875f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 167b45749e0d4747866e5be16306875f, State: Initialized, Role: FOLLOWER
I20260812 05:20:47.547299 11803 consensus_queue.cc:260] T 4b9630239c344b0ab7ec8bc4ff034a62 P 167b45749e0d4747866e5be16306875f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54f92d5624f42a9b11d653eb7bd017c" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 44127 } } peers { permanent_uuid: "167b45749e0d4747866e5be16306875f" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 39085 } } peers { permanent_uuid: "bf52456a845341ebbf4e4e7405844a2d" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 36081 } }
I20260812 05:20:47.547555 11803 ts_tablet_manager.cc:1428] T 4b9630239c344b0ab7ec8bc4ff034a62 P 167b45749e0d4747866e5be16306875f: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260812 05:20:47.547605 11751 heartbeater.cc:503] Master 127.31.100.62:35003 was elected leader, sending a full tablet report...
I20260812 05:20:47.547904 11808 raft_consensus.cc:348] T 4b9630239c344b0ab7ec8bc4ff034a62 P bf52456a845341ebbf4e4e7405844a2d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54f92d5624f42a9b11d653eb7bd017c" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 44127 } } peers { permanent_uuid: "167b45749e0d4747866e5be16306875f" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 39085 } } peers { permanent_uuid: "bf52456a845341ebbf4e4e7405844a2d" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 36081 } }
I20260812 05:20:47.548022 11808 raft_consensus.cc:374] T 4b9630239c344b0ab7ec8bc4ff034a62 P bf52456a845341ebbf4e4e7405844a2d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 05:20:47.548050 11808 raft_consensus.cc:729] T 4b9630239c344b0ab7ec8bc4ff034a62 P bf52456a845341ebbf4e4e7405844a2d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf52456a845341ebbf4e4e7405844a2d, State: Initialized, Role: FOLLOWER
I20260812 05:20:47.548064 11156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:47.544509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.60:37129 (local address 127.31.100.1:56055)
0812 05:20:47.544602 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0812 05:20:47.544613 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0812 05:20:47.544653 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:20:47.545000 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:20:47.545001 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:20:47.545003 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 05:20:47.545066 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:20:47.545067 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:20:47.545965 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:20:47.545966 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:20:47.546840 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:20:47.546843 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:20:47.547824 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:20:47.547825 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:20:47.547907 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:47.547910 (+ 3us) client_negotiation.cc:769] Sending connection context
0812 05:20:47.547962 (+ 52us) client_negotiation.cc:240] Negotiation successful
0812 05:20:47.548001 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":75}
I20260812 05:20:47.548081 11776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:47.544733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.100.1:56055 (local address 127.31.100.60:37129)
0812 05:20:47.544777 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0812 05:20:47.544779 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 05:20:47.544787 (+ 8us) server_negotiation.cc:407] Connection header received
0812 05:20:47.544831 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:20:47.544832 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:20:47.544859 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:20:47.544924 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 05:20:47.545149 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:47.545867 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:47.546964 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:47.547525 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:47.547594 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:47.547676 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0812 05:20:47.547979 (+ 303us) server_negotiation.cc:299] Negotiation successful
0812 05:20:47.548025 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260812 05:20:47.548157 11808 consensus_queue.cc:260] T 4b9630239c344b0ab7ec8bc4ff034a62 P bf52456a845341ebbf4e4e7405844a2d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54f92d5624f42a9b11d653eb7bd017c" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 44127 } } peers { permanent_uuid: "167b45749e0d4747866e5be16306875f" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 39085 } } peers { permanent_uuid: "bf52456a845341ebbf4e4e7405844a2d" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 36081 } }
I20260812 05:20:47.548408 11808 ts_tablet_manager.cc:1428] T 4b9630239c344b0ab7ec8bc4ff034a62 P bf52456a845341ebbf4e4e7405844a2d: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260812 05:20:47.548574 11384 heartbeater.cc:503] Master 127.31.100.62:35003 was elected leader, sending a full tablet report...
I20260812 05:20:47.549026 11151 heartbeater.cc:347] Connected to a master server at 127.31.100.60:37129
I20260812 05:20:47.549082 11151 heartbeater.cc:464] Registering TS with master...
I20260812 05:20:47.549139 11151 heartbeater.cc:511] Master 127.31.100.60:37129 requested a full tablet report, sending...
I20260812 05:20:47.549544 11441 ts_manager.cc:194] Registered new tserver with Master: f54f92d5624f42a9b11d653eb7bd017c (127.31.100.1:44127)
I20260812 05:20:47.551504 11776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:47.548835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.100.2:38819 (local address 127.31.100.60:37129)
0812 05:20:47.549045 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0812 05:20:47.549047 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 05:20:47.549054 (+ 7us) server_negotiation.cc:407] Connection header received
0812 05:20:47.549072 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:20:47.549073 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:20:47.549082 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:20:47.549131 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 05:20:47.549309 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:47.549978 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:47.550901 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:47.551234 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:47.551288 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:47.551294 (+ 6us) server_negotiation.cc:1035] Waiting for connection context
0812 05:20:47.551442 (+ 148us) server_negotiation.cc:299] Negotiation successful
0812 05:20:47.551464 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":189}
I20260812 05:20:47.551513 11389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:47.548648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.60:37129 (local address 127.31.100.2:38819)
0812 05:20:47.548691 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0812 05:20:47.548701 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0812 05:20:47.548732 (+ 31us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:20:47.549173 (+ 441us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:20:47.549174 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:20:47.549176 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 05:20:47.549242 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:20:47.549243 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:20:47.550059 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:20:47.550060 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:20:47.550820 (+ 760us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:20:47.550822 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:20:47.551304 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:20:47.551304 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:20:47.551362 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:47.551365 (+ 3us) client_negotiation.cc:769] Sending connection context
0812 05:20:47.551437 (+ 72us) client_negotiation.cc:240] Negotiation successful
0812 05:20:47.551469 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260812 05:20:47.551787 11382 heartbeater.cc:347] Connected to a master server at 127.31.100.60:37129
I20260812 05:20:47.551831 11382 heartbeater.cc:464] Registering TS with master...
I20260812 05:20:47.551870 11382 heartbeater.cc:511] Master 127.31.100.60:37129 requested a full tablet report, sending...
I20260812 05:20:47.552132 11441 ts_manager.cc:194] Registered new tserver with Master: bf52456a845341ebbf4e4e7405844a2d (127.31.100.2:36081)
W20260812 05:20:47.589815 11387 tablet.cc:2369] T 4b9630239c344b0ab7ec8bc4ff034a62 P bf52456a845341ebbf4e4e7405844a2d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 05:20:47.672454 11155 tablet.cc:2369] T 4b9630239c344b0ab7ec8bc4ff034a62 P f54f92d5624f42a9b11d653eb7bd017c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 05:20:47.682550 11809 raft_consensus.cc:482] T 4b9630239c344b0ab7ec8bc4ff034a62 P f54f92d5624f42a9b11d653eb7bd017c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 05:20:47.682667 11809 raft_consensus.cc:504] T 4b9630239c344b0ab7ec8bc4ff034a62 P f54f92d5624f42a9b11d653eb7bd017c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54f92d5624f42a9b11d653eb7bd017c" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 44127 } } peers { permanent_uuid: "167b45749e0d4747866e5be16306875f" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 39085 } } peers { permanent_uuid: "bf52456a845341ebbf4e4e7405844a2d" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 36081 } }
I20260812 05:20:47.682907 11811 raft_consensus.cc:482] T 4b9630239c344b0ab7ec8bc4ff034a62 P 167b45749e0d4747866e5be16306875f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 05:20:47.683023 11811 raft_consensus.cc:504] T 4b9630239c344b0ab7ec8bc4ff034a62 P 167b45749e0d4747866e5be16306875f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54f92d5624f42a9b11d653eb7bd017c" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 44127 } } peers { permanent_uuid: "167b45749e0d4747866e5be16306875f" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 39085 } } peers { permanent_uuid: "bf52456a845341ebbf4e4e7405844a2d" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 36081 } }
I20260812 05:20:47.683092 11809 leader_election.cc:288] T 4b9630239c344b0ab7ec8bc4ff034a62 P f54f92d5624f42a9b11d653eb7bd017c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 167b45749e0d4747866e5be16306875f (127.31.100.3:39085), bf52456a845341ebbf4e4e7405844a2d (127.31.100.2:36081)
I20260812 05:20:47.683367 11811 leader_election.cc:288] T 4b9630239c344b0ab7ec8bc4ff034a62 P 167b45749e0d4747866e5be16306875f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f54f92d5624f42a9b11d653eb7bd017c (127.31.100.1:44127), bf52456a845341ebbf4e4e7405844a2d (127.31.100.2:36081)
I20260812 05:20:47.686849 11760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:47.683464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.1:44127 (local address 127.31.100.3:43279)
0812 05:20:47.683515 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0812 05:20:47.683524 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0812 05:20:47.683556 (+ 32us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:20:47.683743 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:20:47.683744 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:20:47.683745 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 05:20:47.683817 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:20:47.683819 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:20:47.684685 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:20:47.684686 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:20:47.685881 (+ 1195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:20:47.685884 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:20:47.686603 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:20:47.686604 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:20:47.686672 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:47.686675 (+ 3us) client_negotiation.cc:769] Sending connection context
0812 05:20:47.686716 (+ 41us) client_negotiation.cc:240] Negotiation successful
0812 05:20:47.686754 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260812 05:20:47.686908 11159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:47.683256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.2:36081 (local address 127.31.100.1:37755)
0812 05:20:47.683376 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0812 05:20:47.683387 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0812 05:20:47.683429 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:20:47.683735 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:20:47.683736 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:20:47.683737 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 05:20:47.683803 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:20:47.683805 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:20:47.684740 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:20:47.684741 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:20:47.685920 (+ 1179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:20:47.685922 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:20:47.686668 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:20:47.686669 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:20:47.686755 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:47.686757 (+ 2us) client_negotiation.cc:769] Sending connection context
0812 05:20:47.686809 (+ 52us) client_negotiation.cc:240] Negotiation successful
0812 05:20:47.686840 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"mutex_wait_us":7}
I20260812 05:20:47.686970 11797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:47.683747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.100.1:37361 (local address 127.31.100.3:39085)
0812 05:20:47.683795 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0812 05:20:47.683797 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 05:20:47.683804 (+ 7us) server_negotiation.cc:407] Connection header received
0812 05:20:47.683823 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:20:47.683824 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:20:47.683831 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:20:47.683875 (+ 44us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 05:20:47.684041 (+ 166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:47.684750 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:47.685975 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:47.686529 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:47.686590 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:47.686599 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0812 05:20:47.686869 (+ 270us) server_negotiation.cc:299] Negotiation successful
0812 05:20:47.686906 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260812 05:20:47.687134 11795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:47.683546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.100.3:43279 (local address 127.31.100.1:44127)
0812 05:20:47.683614 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0812 05:20:47.683615 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0812 05:20:47.683622 (+ 7us) server_negotiation.cc:407] Connection header received
0812 05:20:47.683639 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:20:47.683640 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:20:47.683647 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:20:47.683693 (+ 46us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 05:20:47.683912 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:47.684596 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:47.685990 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:47.686506 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:47.686564 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:47.686573 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0812 05:20:47.687038 (+ 465us) server_negotiation.cc:299] Negotiation successful
0812 05:20:47.687079 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260812 05:20:47.687155 11759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:47.683623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.2:36081 (local address 127.31.100.3:57599)
0812 05:20:47.683653 (+ 30us) negotiation.cc:107] Waiting for socket to connect
0812 05:20:47.683660 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0812 05:20:47.683688 (+ 28us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:20:47.684025 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:20:47.684026 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:20:47.684028 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 05:20:47.684084 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:20:47.684085 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:20:47.685083 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:20:47.685084 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:20:47.686165 (+ 1081us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:20:47.686167 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:20:47.686943 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:20:47.686944 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:20:47.687032 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:47.687035 (+ 3us) client_negotiation.cc:769] Sending connection context
0812 05:20:47.687063 (+ 28us) client_negotiation.cc:240] Negotiation successful
0812 05:20:47.687094 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18,"mutex_wait_us":1}
I20260812 05:20:47.687227 11798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:47.683499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.100.1:37755 (local address 127.31.100.2:36081)
0812 05:20:47.683586 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0812 05:20:47.683587 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0812 05:20:47.683596 (+ 9us) server_negotiation.cc:407] Connection header received
0812 05:20:47.683625 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:20:47.683626 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:20:47.683636 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:20:47.683687 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 05:20:47.683877 (+ 190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:47.684653 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:47.686010 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:47.686567 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:47.686617 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:47.686624 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0812 05:20:47.687116 (+ 492us) server_negotiation.cc:299] Negotiation successful
0812 05:20:47.687158 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260812 05:20:47.687494 11070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b9630239c344b0ab7ec8bc4ff034a62" candidate_uuid: "167b45749e0d4747866e5be16306875f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f54f92d5624f42a9b11d653eb7bd017c" is_pre_election: true
I20260812 05:20:47.687593 11070 raft_consensus.cc:2454] T 4b9630239c344b0ab7ec8bc4ff034a62 P f54f92d5624f42a9b11d653eb7bd017c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 167b45749e0d4747866e5be16306875f in term 0.
I20260812 05:20:47.687536 11309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b9630239c344b0ab7ec8bc4ff034a62" candidate_uuid: "f54f92d5624f42a9b11d653eb7bd017c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf52456a845341ebbf4e4e7405844a2d" is_pre_election: true
I20260812 05:20:47.687677 11814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:47.683718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.100.3:57599 (local address 127.31.100.2:36081)
0812 05:20:47.683884 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0812 05:20:47.683886 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 05:20:47.683893 (+ 7us) server_negotiation.cc:407] Connection header received
0812 05:20:47.683914 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:20:47.683914 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:20:47.683923 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:20:47.684140 (+ 217us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 05:20:47.684174 (+ 34us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:47.684950 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:47.686286 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:47.686855 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:47.687337 (+ 482us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:47.687348 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0812 05:20:47.687374 (+ 26us) server_negotiation.cc:299] Negotiation successful
0812 05:20:47.687445 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":94,"threads_started":1}
I20260812 05:20:47.687749 11309 raft_consensus.cc:2454] T 4b9630239c344b0ab7ec8bc4ff034a62 P bf52456a845341ebbf4e4e7405844a2d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f54f92d5624f42a9b11d653eb7bd017c in term 0.
I20260812 05:20:47.687773 11553 leader_election.cc:302] T 4b9630239c344b0ab7ec8bc4ff034a62 P 167b45749e0d4747866e5be16306875f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 167b45749e0d4747866e5be16306875f, f54f92d5624f42a9b11d653eb7bd017c; no voters:
I20260812 05:20:47.687891 11156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:47.683247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.3:39085 (local address 127.31.100.1:37361)
0812 05:20:47.683307 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0812 05:20:47.683319 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0812 05:20:47.683399 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:20:47.683913 (+ 514us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:20:47.683914 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:20:47.683916 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 05:20:47.683973 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:20:47.683974 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:20:47.684823 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:20:47.684824 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:20:47.685876 (+ 1052us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:20:47.685879 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:20:47.686656 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:20:47.686657 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:20:47.686714 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:47.686717 (+ 3us) client_negotiation.cc:769] Sending connection context
0812 05:20:47.686760 (+ 43us) client_negotiation.cc:240] Negotiation successful
0812 05:20:47.686799 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41,"mutex_wait_us":2}
I20260812 05:20:47.687866 11307 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b9630239c344b0ab7ec8bc4ff034a62" candidate_uuid: "167b45749e0d4747866e5be16306875f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf52456a845341ebbf4e4e7405844a2d" is_pre_election: true
I20260812 05:20:47.687892 11811 raft_consensus.cc:2781] T 4b9630239c344b0ab7ec8bc4ff034a62 P 167b45749e0d4747866e5be16306875f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 05:20:47.687935 11811 raft_consensus.cc:482] T 4b9630239c344b0ab7ec8bc4ff034a62 P 167b45749e0d4747866e5be16306875f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 05:20:47.687932 11307 raft_consensus.cc:2454] T 4b9630239c344b0ab7ec8bc4ff034a62 P bf52456a845341ebbf4e4e7405844a2d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 167b45749e0d4747866e5be16306875f in term 0.
I20260812 05:20:47.687958 11811 raft_consensus.cc:3037] T 4b9630239c344b0ab7ec8bc4ff034a62 P 167b45749e0d4747866e5be16306875f [term 0 FOLLOWER]: Advancing to term 1
I20260812 05:20:47.687968 10972 leader_election.cc:302] T 4b9630239c344b0ab7ec8bc4ff034a62 P f54f92d5624f42a9b11d653eb7bd017c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bf52456a845341ebbf4e4e7405844a2d, f54f92d5624f42a9b11d653eb7bd017c; no voters:
I20260812 05:20:47.688117 11809 raft_consensus.cc:2781] T 4b9630239c344b0ab7ec8bc4ff034a62 P f54f92d5624f42a9b11d653eb7bd017c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 05:20:47.688177 11809 raft_consensus.cc:482] T 4b9630239c344b0ab7ec8bc4ff034a62 P f54f92d5624f42a9b11d653eb7bd017c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 05:20:47.688207 11809 raft_consensus.cc:3037] T 4b9630239c344b0ab7ec8bc4ff034a62 P f54f92d5624f42a9b11d653eb7bd017c [term 0 FOLLOWER]: Advancing to term 1
I20260812 05:20:47.688408 11673 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b9630239c344b0ab7ec8bc4ff034a62" candidate_uuid: "f54f92d5624f42a9b11d653eb7bd017c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "167b45749e0d4747866e5be16306875f" is_pre_election: true
I20260812 05:20:47.688889 11811 raft_consensus.cc:504] T 4b9630239c344b0ab7ec8bc4ff034a62 P 167b45749e0d4747866e5be16306875f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54f92d5624f42a9b11d653eb7bd017c" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 44127 } } peers { permanent_uuid: "167b45749e0d4747866e5be16306875f" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 39085 } } peers { permanent_uuid: "bf52456a845341ebbf4e4e7405844a2d" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 36081 } }
I20260812 05:20:47.688911 11809 raft_consensus.cc:504] T 4b9630239c344b0ab7ec8bc4ff034a62 P f54f92d5624f42a9b11d653eb7bd017c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54f92d5624f42a9b11d653eb7bd017c" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 44127 } } peers { permanent_uuid: "167b45749e0d4747866e5be16306875f" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 39085 } } peers { permanent_uuid: "bf52456a845341ebbf4e4e7405844a2d" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 36081 } }
I20260812 05:20:47.689020 11809 leader_election.cc:288] T 4b9630239c344b0ab7ec8bc4ff034a62 P f54f92d5624f42a9b11d653eb7bd017c [CANDIDATE]: Term 1 election: Requested vote from peers 167b45749e0d4747866e5be16306875f (127.31.100.3:39085), bf52456a845341ebbf4e4e7405844a2d (127.31.100.2:36081)
I20260812 05:20:47.689021 11673 raft_consensus.cc:2379] T 4b9630239c344b0ab7ec8bc4ff034a62 P 167b45749e0d4747866e5be16306875f [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f54f92d5624f42a9b11d653eb7bd017c in current term 1: Already voted for candidate 167b45749e0d4747866e5be16306875f in this term.
I20260812 05:20:47.689131 11070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b9630239c344b0ab7ec8bc4ff034a62" candidate_uuid: "167b45749e0d4747866e5be16306875f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f54f92d5624f42a9b11d653eb7bd017c"
I20260812 05:20:47.689175 11673 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b9630239c344b0ab7ec8bc4ff034a62" candidate_uuid: "f54f92d5624f42a9b11d653eb7bd017c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "167b45749e0d4747866e5be16306875f"
I20260812 05:20:47.689213 11070 raft_consensus.cc:2379] T 4b9630239c344b0ab7ec8bc4ff034a62 P f54f92d5624f42a9b11d653eb7bd017c [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 167b45749e0d4747866e5be16306875f in current term 1: Already voted for candidate f54f92d5624f42a9b11d653eb7bd017c in this term.
I20260812 05:20:47.689229 11673 raft_consensus.cc:2379] T 4b9630239c344b0ab7ec8bc4ff034a62 P 167b45749e0d4747866e5be16306875f [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate f54f92d5624f42a9b11d653eb7bd017c in current term 1: Already voted for candidate 167b45749e0d4747866e5be16306875f in this term.
I20260812 05:20:47.689275 11307 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b9630239c344b0ab7ec8bc4ff034a62" candidate_uuid: "f54f92d5624f42a9b11d653eb7bd017c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf52456a845341ebbf4e4e7405844a2d"
I20260812 05:20:47.689349 11307 raft_consensus.cc:3037] T 4b9630239c344b0ab7ec8bc4ff034a62 P bf52456a845341ebbf4e4e7405844a2d [term 0 FOLLOWER]: Advancing to term 1
I20260812 05:20:47.689435 11811 leader_election.cc:288] T 4b9630239c344b0ab7ec8bc4ff034a62 P 167b45749e0d4747866e5be16306875f [CANDIDATE]: Term 1 election: Requested vote from peers f54f92d5624f42a9b11d653eb7bd017c (127.31.100.1:44127), bf52456a845341ebbf4e4e7405844a2d (127.31.100.2:36081)
I20260812 05:20:47.689590 11309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b9630239c344b0ab7ec8bc4ff034a62" candidate_uuid: "167b45749e0d4747866e5be16306875f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf52456a845341ebbf4e4e7405844a2d"
I20260812 05:20:47.689821 11547 leader_election.cc:302] T 4b9630239c344b0ab7ec8bc4ff034a62 P 167b45749e0d4747866e5be16306875f [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 167b45749e0d4747866e5be16306875f; no voters: bf52456a845341ebbf4e4e7405844a2d, f54f92d5624f42a9b11d653eb7bd017c
I20260812 05:20:47.689918 11811 raft_consensus.cc:2726] T 4b9630239c344b0ab7ec8bc4ff034a62 P 167b45749e0d4747866e5be16306875f [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260812 05:20:47.690272 11307 raft_consensus.cc:2454] T 4b9630239c344b0ab7ec8bc4ff034a62 P bf52456a845341ebbf4e4e7405844a2d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f54f92d5624f42a9b11d653eb7bd017c in term 1.
I20260812 05:20:47.690426 10972 leader_election.cc:302] T 4b9630239c344b0ab7ec8bc4ff034a62 P f54f92d5624f42a9b11d653eb7bd017c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: bf52456a845341ebbf4e4e7405844a2d, f54f92d5624f42a9b11d653eb7bd017c; no voters: 167b45749e0d4747866e5be16306875f
I20260812 05:20:47.690546 11809 raft_consensus.cc:2781] T 4b9630239c344b0ab7ec8bc4ff034a62 P f54f92d5624f42a9b11d653eb7bd017c [term 1 FOLLOWER]: Leader election won for term 1
I20260812 05:20:47.690663 11809 raft_consensus.cc:686] T 4b9630239c344b0ab7ec8bc4ff034a62 P f54f92d5624f42a9b11d653eb7bd017c [term 1 LEADER]: Becoming Leader. State: Replica: f54f92d5624f42a9b11d653eb7bd017c, State: Running, Role: LEADER
I20260812 05:20:47.690745 11809 consensus_queue.cc:237] T 4b9630239c344b0ab7ec8bc4ff034a62 P f54f92d5624f42a9b11d653eb7bd017c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54f92d5624f42a9b11d653eb7bd017c" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 44127 } } peers { permanent_uuid: "167b45749e0d4747866e5be16306875f" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 39085 } } peers { permanent_uuid: "bf52456a845341ebbf4e4e7405844a2d" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 36081 } }
I20260812 05:20:47.691548 9738 catalog_manager.cc:5495] T 4b9630239c344b0ab7ec8bc4ff034a62 P f54f92d5624f42a9b11d653eb7bd017c reported cstate change: term changed from 0 to 1, leader changed from <none> to f54f92d5624f42a9b11d653eb7bd017c (127.31.100.1). New cstate: current_term: 1 leader_uuid: "f54f92d5624f42a9b11d653eb7bd017c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54f92d5624f42a9b11d653eb7bd017c" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 44127 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "167b45749e0d4747866e5be16306875f" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 39085 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bf52456a845341ebbf4e4e7405844a2d" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 36081 } health_report { overall_health: UNKNOWN } } }
I20260812 05:20:47.706426 11789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:47.704107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.60:37129 (local address 127.0.0.1:55774)
0812 05:20:47.704158 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0812 05:20:47.704167 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0812 05:20:47.704209 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:20:47.704323 (+ 114us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:20:47.704324 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:20:47.704325 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 05:20:47.704366 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:20:47.704368 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:20:47.704945 (+ 577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:20:47.704946 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:20:47.705673 (+ 727us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:20:47.705675 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:20:47.706223 (+ 548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:20:47.706224 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:20:47.706288 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:47.706291 (+ 3us) client_negotiation.cc:769] Sending connection context
0812 05:20:47.706337 (+ 46us) client_negotiation.cc:240] Negotiation successful
0812 05:20:47.706372 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260812 05:20:47.706465 11776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:47.704180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55774 (local address 127.31.100.60:37129)
0812 05:20:47.704230 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0812 05:20:47.704232 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 05:20:47.704239 (+ 7us) server_negotiation.cc:407] Connection header received
0812 05:20:47.704261 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:20:47.704261 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:20:47.704273 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:20:47.704315 (+ 42us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 05:20:47.704411 (+ 96us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:47.704886 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:47.705744 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:47.706094 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:47.706142 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:47.706149 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0812 05:20:47.706356 (+ 207us) server_negotiation.cc:299] Negotiation successful
0812 05:20:47.706416 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260812 05:20:47.706806 11458 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 22c6383cda19494f95cfee62ca4d0ccf [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.10] Dedup: 1.7->[1.8-1.10]
I20260812 05:20:47.707688 11502 sys_catalog.cc:455] T 00000000000000000000000000000000 P 22c6383cda19494f95cfee62ca4d0ccf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 070ed78c702d4333919d5d31da7b7608. Latest consensus state: current_term: 1 leader_uuid: "070ed78c702d4333919d5d31da7b7608" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "070ed78c702d4333919d5d31da7b7608" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 35003 } } peers { permanent_uuid: "e8be870d7089424d82f5be0171ba5242" member_type: VOTER last_known_addr { host: "127.31.100.61" port: 34209 } attrs { promote: false } } peers { permanent_uuid: "22c6383cda19494f95cfee62ca4d0ccf" member_type: NON_VOTER last_known_addr { host: "127.31.100.60" port: 37129 } attrs { promote: true } } }
I20260812 05:20:47.707788 11502 sys_catalog.cc:458] T 00000000000000000000000000000000 P 22c6383cda19494f95cfee62ca4d0ccf [sys.catalog]: This master's current role is: LEARNER
I20260812 05:20:47.712551 11795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:47.710256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60834 (local address 127.31.100.1:44127)
0812 05:20:47.710303 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0812 05:20:47.710305 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 05:20:47.710312 (+ 7us) server_negotiation.cc:407] Connection header received
0812 05:20:47.710348 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:20:47.710348 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:20:47.710374 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:20:47.710433 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 05:20:47.710540 (+ 107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:47.711025 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:47.711740 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:47.711997 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:47.712059 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:47.712215 (+ 156us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 05:20:47.712349 (+ 134us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 05:20:47.712424 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0812 05:20:47.712470 (+ 46us) server_negotiation.cc:299] Negotiation successful
0812 05:20:47.712491 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260812 05:20:47.714154 11673 raft_consensus.cc:1261] T 4b9630239c344b0ab7ec8bc4ff034a62 P 167b45749e0d4747866e5be16306875f [term 1 FOLLOWER]: Refusing update from remote peer f54f92d5624f42a9b11d653eb7bd017c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 05:20:47.714222 11307 raft_consensus.cc:1261] T 4b9630239c344b0ab7ec8bc4ff034a62 P bf52456a845341ebbf4e4e7405844a2d [term 1 FOLLOWER]: Refusing update from remote peer f54f92d5624f42a9b11d653eb7bd017c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 05:20:47.714593 11815 consensus_queue.cc:1035] T 4b9630239c344b0ab7ec8bc4ff034a62 P f54f92d5624f42a9b11d653eb7bd017c [LEADER]: Connected to new peer: Peer: permanent_uuid: "167b45749e0d4747866e5be16306875f" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 39085 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 05:20:47.714707 11809 consensus_queue.cc:1035] T 4b9630239c344b0ab7ec8bc4ff034a62 P f54f92d5624f42a9b11d653eb7bd017c [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf52456a845341ebbf4e4e7405844a2d" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 36081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 05:20:47.719179 11824 mvcc.cc:205] Tried to move back new op lower bound from 7317553347435081728 to 7317553347341856768. Current Snapshot: MvccSnapshot[applied={T|T < 7317553347435081728 or (T in {7317553347435081728})}]
I20260812 05:20:47.720335 9738 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48228:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 05:20:47.720423 9738 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48228:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 05:20:47.720960 10936 sys_catalog.cc:455] T 00000000000000000000000000000000 P 070ed78c702d4333919d5d31da7b7608 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "070ed78c702d4333919d5d31da7b7608" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "070ed78c702d4333919d5d31da7b7608" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 35003 } } peers { permanent_uuid: "e8be870d7089424d82f5be0171ba5242" member_type: VOTER last_known_addr { host: "127.31.100.61" port: 34209 } attrs { promote: false } } peers { permanent_uuid: "22c6383cda19494f95cfee62ca4d0ccf" member_type: NON_VOTER last_known_addr { host: "127.31.100.60" port: 37129 } attrs { promote: true } } }
I20260812 05:20:47.721055 10936 sys_catalog.cc:458] T 00000000000000000000000000000000 P 070ed78c702d4333919d5d31da7b7608 [sys.catalog]: This master's current role is: LEADER
I20260812 05:20:47.721700 11834 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 070ed78c702d4333919d5d31da7b7608: attempting to promote NON_VOTER 22c6383cda19494f95cfee62ca4d0ccf to VOTER
I20260812 05:20:47.721827 11834 consensus_queue.cc:237] T 00000000000000000000000000000000 P 070ed78c702d4333919d5d31da7b7608 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "070ed78c702d4333919d5d31da7b7608" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 35003 } } peers { permanent_uuid: "e8be870d7089424d82f5be0171ba5242" member_type: VOTER last_known_addr { host: "127.31.100.61" port: 34209 } attrs { promote: false } } peers { permanent_uuid: "22c6383cda19494f95cfee62ca4d0ccf" member_type: VOTER last_known_addr { host: "127.31.100.60" port: 37129 } attrs { promote: false } }
I20260812 05:20:47.721868 9738 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 070ed78c702d4333919d5d31da7b7608: Sending DeleteTablet for 3 replicas of tablet 4b9630239c344b0ab7ec8bc4ff034a62
I20260812 05:20:47.722167 11630 tablet_service.cc:1514] Processing DeleteTablet for tablet 4b9630239c344b0ab7ec8bc4ff034a62 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 05:20:47 UTC) from {username='oryx', principal='oryx/127.31.100.62@KRBTEST.COM'} at 127.0.0.1:44782
I20260812 05:20:47.722254 11458 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 22c6383cda19494f95cfee62ca4d0ccf [term 1 LEARNER]: Refusing update from remote peer 070ed78c702d4333919d5d31da7b7608: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20260812 05:20:47.722250 11275 tablet_service.cc:1514] Processing DeleteTablet for tablet 4b9630239c344b0ab7ec8bc4ff034a62 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 05:20:47 UTC) from {username='oryx', principal='oryx/127.31.100.62@KRBTEST.COM'} at 127.0.0.1:41760
I20260812 05:20:47.722425 10936 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 070ed78c702d4333919d5d31da7b7608 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22c6383cda19494f95cfee62ca4d0ccf" member_type: VOTER last_known_addr { host: "127.31.100.60" port: 37129 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20260812 05:20:47.722473 11840 tablet_replica.cc:330] T 4b9630239c344b0ab7ec8bc4ff034a62 P bf52456a845341ebbf4e4e7405844a2d: stopping tablet replica
I20260812 05:20:47.722563 11840 raft_consensus.cc:2229] T 4b9630239c344b0ab7ec8bc4ff034a62 P bf52456a845341ebbf4e4e7405844a2d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 05:20:47.722675 11840 raft_consensus.cc:2258] T 4b9630239c344b0ab7ec8bc4ff034a62 P bf52456a845341ebbf4e4e7405844a2d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 05:20:47.723136 11840 ts_tablet_manager.cc:1905] T 4b9630239c344b0ab7ec8bc4ff034a62 P bf52456a845341ebbf4e4e7405844a2d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 05:20:47.723320 32144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0waFvT/build/release/bin/kudu with pid 9687
I20260812 05:20:47.732892 11840 ts_tablet_manager.cc:1918] T 4b9630239c344b0ab7ec8bc4ff034a62 P bf52456a845341ebbf4e4e7405844a2d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 05:20:47.732991 11840 log.cc:1196] T 4b9630239c344b0ab7ec8bc4ff034a62 P bf52456a845341ebbf4e4e7405844a2d: Deleting WAL directory at /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786512017613366-32144-0/minicluster-data/ts-1/wal/wals/4b9630239c344b0ab7ec8bc4ff034a62
I20260812 05:20:47.733253 11840 ts_tablet_manager.cc:1939] T 4b9630239c344b0ab7ec8bc4ff034a62 P bf52456a845341ebbf4e4e7405844a2d: Deleting consensus metadata
W20260812 05:20:47.737511 11418 connection.cc:535] server connection from 127.0.0.1:55774 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260812 05:20:47.737517 11194 connection.cc:535] server connection from 127.0.0.1:41760 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260812 05:20:47.737743 11043 tablet_service.cc:1514] Processing DeleteTablet for tablet 4b9630239c344b0ab7ec8bc4ff034a62 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 05:20:47 UTC) from {username='oryx', principal='oryx/127.31.100.62@KRBTEST.COM'} at 127.0.0.1:60818
I20260812 05:20:47.737972 11844 tablet_replica.cc:330] T 4b9630239c344b0ab7ec8bc4ff034a62 P f54f92d5624f42a9b11d653eb7bd017c: stopping tablet replica
I20260812 05:20:47.738088 11844 raft_consensus.cc:2229] T 4b9630239c344b0ab7ec8bc4ff034a62 P f54f92d5624f42a9b11d653eb7bd017c [term 1 LEADER]: Raft consensus shutting down.
I20260812 05:20:47.738184 11844 raft_consensus.cc:2258] T 4b9630239c344b0ab7ec8bc4ff034a62 P f54f92d5624f42a9b11d653eb7bd017c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 05:20:47.738181 10715 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e8be870d7089424d82f5be0171ba5242 [term 1 FOLLOWER]: Refusing update from remote peer 070ed78c702d4333919d5d31da7b7608: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
W20260812 05:20:47.738665 10673 connection.cc:535] server connection from 127.0.0.1:49854 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260812 05:20:47.738679 11844 ts_tablet_manager.cc:1905] T 4b9630239c344b0ab7ec8bc4ff034a62 P f54f92d5624f42a9b11d653eb7bd017c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 05:20:47.739373 11838 tablet_replica.cc:330] T 4b9630239c344b0ab7ec8bc4ff034a62 P 167b45749e0d4747866e5be16306875f: stopping tablet replica
I20260812 05:20:47.739506 11838 raft_consensus.cc:2229] T 4b9630239c344b0ab7ec8bc4ff034a62 P 167b45749e0d4747866e5be16306875f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 05:20:47.739622 11838 raft_consensus.cc:2258] T 4b9630239c344b0ab7ec8bc4ff034a62 P 167b45749e0d4747866e5be16306875f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 05:20:47.740113 11838 ts_tablet_manager.cc:1905] T 4b9630239c344b0ab7ec8bc4ff034a62 P 167b45749e0d4747866e5be16306875f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 05:20:47.740764 11844 ts_tablet_manager.cc:1918] T 4b9630239c344b0ab7ec8bc4ff034a62 P f54f92d5624f42a9b11d653eb7bd017c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 05:20:47.740937 11844 log.cc:1196] T 4b9630239c344b0ab7ec8bc4ff034a62 P f54f92d5624f42a9b11d653eb7bd017c: Deleting WAL directory at /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786512017613366-32144-0/minicluster-data/ts-0/wal/wals/4b9630239c344b0ab7ec8bc4ff034a62
I20260812 05:20:47.741263 11844 ts_tablet_manager.cc:1939] T 4b9630239c344b0ab7ec8bc4ff034a62 P f54f92d5624f42a9b11d653eb7bd017c: Deleting consensus metadata
W20260812 05:20:47.741633 10972 connection.cc:441] server connection from 127.0.0.1:60818 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60818 (request call id 1) could send its response
I20260812 05:20:47.741730 11838 ts_tablet_manager.cc:1918] T 4b9630239c344b0ab7ec8bc4ff034a62 P 167b45749e0d4747866e5be16306875f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 05:20:47.741816 11838 log.cc:1196] T 4b9630239c344b0ab7ec8bc4ff034a62 P 167b45749e0d4747866e5be16306875f: Deleting WAL directory at /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786512017613366-32144-0/minicluster-data/ts-2/wal/wals/4b9630239c344b0ab7ec8bc4ff034a62
I20260812 05:20:47.742062 11838 ts_tablet_manager.cc:1939] T 4b9630239c344b0ab7ec8bc4ff034a62 P 167b45749e0d4747866e5be16306875f: Deleting consensus metadata
W20260812 05:20:47.742357 11553 connection.cc:441] server connection from 127.0.0.1:44782 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44782 (request call id 1) could send its response
I20260812 05:20:47.748797 10893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:47.740488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49896 (local address 127.31.100.61:34209)
0812 05:20:47.740527 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0812 05:20:47.740530 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 05:20:47.741126 (+ 596us) server_negotiation.cc:407] Connection header received
0812 05:20:47.741240 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:20:47.741241 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:20:47.741252 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:20:47.741278 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:20:47.742125 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:47.742886 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:47.743920 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:47.744194 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:47.744233 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:47.746008 (+ 1775us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 05:20:47.746010 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 05:20:47.746010 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 05:20:47.746017 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 05:20:47.747560 (+ 1543us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:20:47.747955 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:20:47.747956 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:20:47.747957 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:20:47.748009 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:20:47.748203 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:20:47.748204 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:20:47.748205 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:20:47.748331 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 05:20:47.748421 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0812 05:20:47.748644 (+ 223us) server_negotiation.cc:299] Negotiation successful
0812 05:20:47.748710 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260812 05:20:47.749748 11773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:47.722946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49888 (local address 127.31.100.61:34209)
0812 05:20:47.722983 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0812 05:20:47.722985 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 05:20:47.742440 (+ 19455us) server_negotiation.cc:407] Connection header received
0812 05:20:47.742511 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:20:47.742512 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:20:47.742526 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:20:47.742575 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:20:47.743143 (+ 568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:47.743849 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:47.744727 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:47.744932 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:47.744983 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:47.746467 (+ 1484us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 05:20:47.746469 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 05:20:47.746469 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 05:20:47.746476 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 05:20:47.748273 (+ 1797us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:20:47.748838 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:20:47.748839 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:20:47.748840 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:20:47.748890 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:20:47.749201 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:20:47.749202 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:20:47.749202 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:20:47.749314 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 05:20:47.749376 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0812 05:20:47.749588 (+ 212us) server_negotiation.cc:299] Negotiation successful
0812 05:20:47.749668 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260812 05:20:47.751288 11761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:47.740773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55782 (local address 127.31.100.60:37129)
0812 05:20:47.740824 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0812 05:20:47.740825 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0812 05:20:47.742987 (+ 2162us) server_negotiation.cc:407] Connection header received
0812 05:20:47.743013 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:20:47.743014 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:20:47.743027 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:20:47.743073 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:20:47.744386 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:47.745493 (+ 1107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:47.746503 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:47.746820 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:47.746857 (+ 37us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:47.748146 (+ 1289us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 05:20:47.748147 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 05:20:47.748148 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 05:20:47.748155 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 05:20:47.750060 (+ 1905us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:20:47.750574 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:20:47.750575 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:20:47.750575 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:20:47.750635 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:20:47.750818 (+ 183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:20:47.750818 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:20:47.750818 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:20:47.750929 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 05:20:47.751015 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0812 05:20:47.751120 (+ 105us) server_negotiation.cc:299] Negotiation successful
0812 05:20:47.751200 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260812 05:20:47.755570 11776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:47.723219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55778 (local address 127.31.100.60:37129)
0812 05:20:47.723249 (+ 30us) server_negotiation.cc:206] Beginning negotiation
0812 05:20:47.723250 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0812 05:20:47.749780 (+ 26530us) server_negotiation.cc:407] Connection header received
0812 05:20:47.749854 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:20:47.749855 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:20:47.749865 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:20:47.749909 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:20:47.750358 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:47.750963 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:47.751969 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:47.752266 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:47.752330 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:47.753273 (+ 943us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 05:20:47.753274 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 05:20:47.753274 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 05:20:47.753281 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 05:20:47.754610 (+ 1329us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:20:47.754935 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:20:47.754936 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:20:47.754936 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:20:47.755000 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:20:47.755148 (+ 148us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:20:47.755148 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:20:47.755149 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:20:47.755263 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 05:20:47.755348 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0812 05:20:47.755407 (+ 59us) server_negotiation.cc:299] Negotiation successful
0812 05:20:47.755483 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14}
I20260812 05:20:48.719287 11858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:48.719037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.62:35003 (local address 127.31.100.1:44221)
0812 05:20:48.719184 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0812 05:20:48.719232 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.100.62:35003: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":66,"threads_started":1}
I20260812 05:20:48.719286 11856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:48.719039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.62:35003 (local address 127.31.100.2:53573)
0812 05:20:48.719172 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0812 05:20:48.719232 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.100.62:35003: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":76,"threads_started":1}
I20260812 05:20:48.720077 11859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:48.719857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.62:35003 (local address 127.31.100.3:45369)
0812 05:20:48.719950 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0812 05:20:48.720031 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.100.62:35003: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77,"thread_start_us":51,"threads_started":1}
W20260812 05:20:48.720266 11751 heartbeater.cc:644] Failed to heartbeat to 127.31.100.62:35003 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.31.100.62:35003: connect: Connection refused (error 111)
I20260812 05:20:49.100446 11861 raft_consensus.cc:482] T 00000000000000000000000000000000 P 22c6383cda19494f95cfee62ca4d0ccf [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 070ed78c702d4333919d5d31da7b7608)
I20260812 05:20:49.100595 11861 raft_consensus.cc:504] T 00000000000000000000000000000000 P 22c6383cda19494f95cfee62ca4d0ccf [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "070ed78c702d4333919d5d31da7b7608" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 35003 } } peers { permanent_uuid: "e8be870d7089424d82f5be0171ba5242" member_type: VOTER last_known_addr { host: "127.31.100.61" port: 34209 } attrs { promote: false } } peers { permanent_uuid: "22c6383cda19494f95cfee62ca4d0ccf" member_type: VOTER last_known_addr { host: "127.31.100.60" port: 37129 } attrs { promote: false } }
I20260812 05:20:49.100878 11861 leader_election.cc:288] T 00000000000000000000000000000000 P 22c6383cda19494f95cfee62ca4d0ccf [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 070ed78c702d4333919d5d31da7b7608 (127.31.100.62:35003), e8be870d7089424d82f5be0171ba5242 (127.31.100.61:34209)
I20260812 05:20:49.101240 11868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:49.101005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.62:35003 (local address 127.0.0.1:48764)
0812 05:20:49.101133 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0812 05:20:49.101192 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.100.62:35003: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":48,"threads_started":1}
W20260812 05:20:49.101410 11416 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.100.62:35003: connect: Connection refused (error 111)
I20260812 05:20:49.101797 11868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:49.101707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.62:35003 (local address 127.0.0.1:48780)
0812 05:20:49.101749 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0812 05:20:49.101774 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.100.62:35003: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260812 05:20:49.101881 11416 leader_election.cc:334] T 00000000000000000000000000000000 P 22c6383cda19494f95cfee62ca4d0ccf [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 070ed78c702d4333919d5d31da7b7608 (127.31.100.62:35003): Network error: Client connection negotiation failed: client connection to 127.31.100.62:35003: connect: Connection refused (error 111)
I20260812 05:20:49.103951 11870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:49.101266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49900 (local address 127.31.100.61:34209)
0812 05:20:49.101401 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0812 05:20:49.101403 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 05:20:49.101410 (+ 7us) server_negotiation.cc:407] Connection header received
0812 05:20:49.101463 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:20:49.101464 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:20:49.101493 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:20:49.101575 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 05:20:49.101708 (+ 133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:49.102287 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:49.103228 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:49.103640 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:49.103704 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:49.103783 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0812 05:20:49.103844 (+ 61us) server_negotiation.cc:299] Negotiation successful
0812 05:20:49.103884 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":69,"threads_started":1}
I20260812 05:20:49.103971 11869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:49.101167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.61:34209 (local address 127.0.0.1:49900)
0812 05:20:49.101272 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0812 05:20:49.101284 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0812 05:20:49.101358 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:20:49.101582 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:20:49.101583 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:20:49.101585 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 05:20:49.101646 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:20:49.101648 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:20:49.102364 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:20:49.102365 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:20:49.103142 (+ 777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:20:49.103144 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:20:49.103715 (+ 571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:20:49.103716 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:20:49.103779 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:49.103783 (+ 4us) client_negotiation.cc:769] Sending connection context
0812 05:20:49.103838 (+ 55us) client_negotiation.cc:240] Negotiation successful
0812 05:20:49.103882 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":86,"thread_start_us":58,"threads_started":1}
I20260812 05:20:49.104274 10715 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "22c6383cda19494f95cfee62ca4d0ccf" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "e8be870d7089424d82f5be0171ba5242" is_pre_election: true
I20260812 05:20:49.104537 11416 leader_election.cc:302] T 00000000000000000000000000000000 P 22c6383cda19494f95cfee62ca4d0ccf [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22c6383cda19494f95cfee62ca4d0ccf; no voters: 070ed78c702d4333919d5d31da7b7608, e8be870d7089424d82f5be0171ba5242
I20260812 05:20:49.104676 11861 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 22c6383cda19494f95cfee62ca4d0ccf [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260812 05:20:49.238327 11876 raft_consensus.cc:482] T 00000000000000000000000000000000 P e8be870d7089424d82f5be0171ba5242 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 070ed78c702d4333919d5d31da7b7608)
I20260812 05:20:49.238487 11876 raft_consensus.cc:504] T 00000000000000000000000000000000 P e8be870d7089424d82f5be0171ba5242 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "070ed78c702d4333919d5d31da7b7608" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 35003 } } peers { permanent_uuid: "e8be870d7089424d82f5be0171ba5242" member_type: VOTER last_known_addr { host: "127.31.100.61" port: 34209 } attrs { promote: false } } peers { permanent_uuid: "22c6383cda19494f95cfee62ca4d0ccf" member_type: NON_VOTER last_known_addr { host: "127.31.100.60" port: 37129 } attrs { promote: true } }
I20260812 05:20:49.238729 11876 leader_election.cc:288] T 00000000000000000000000000000000 P e8be870d7089424d82f5be0171ba5242 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 070ed78c702d4333919d5d31da7b7608 (127.31.100.62:35003)
I20260812 05:20:49.239079 11878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:49.238865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.62:35003 (local address 127.0.0.1:48820)
0812 05:20:49.238970 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0812 05:20:49.239024 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.100.62:35003: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86,"thread_start_us":46,"threads_started":1}
W20260812 05:20:49.239246 10673 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.100.62:35003: connect: Connection refused (error 111)
I20260812 05:20:49.239634 11878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:49.239546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.62:35003 (local address 127.0.0.1:48832)
0812 05:20:49.239588 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0812 05:20:49.239614 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.100.62:35003: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260812 05:20:49.239725 10673 leader_election.cc:334] T 00000000000000000000000000000000 P e8be870d7089424d82f5be0171ba5242 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 070ed78c702d4333919d5d31da7b7608 (127.31.100.62:35003): Network error: Client connection negotiation failed: client connection to 127.31.100.62:35003: connect: Connection refused (error 111)
I20260812 05:20:49.239781 10673 leader_election.cc:302] T 00000000000000000000000000000000 P e8be870d7089424d82f5be0171ba5242 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: e8be870d7089424d82f5be0171ba5242; no voters: 070ed78c702d4333919d5d31da7b7608
I20260812 05:20:49.239857 11876 raft_consensus.cc:2726] T 00000000000000000000000000000000 P e8be870d7089424d82f5be0171ba5242 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260812 05:20:49.720079 11880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:49.719855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.62:35003 (local address 127.31.100.2:42103)
0812 05:20:49.719978 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0812 05:20:49.720022 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.100.62:35003: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":78,"threads_started":1}
I20260812 05:20:49.720093 11882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:49.719808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.62:35003 (local address 127.31.100.1:43631)
0812 05:20:49.720015 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0812 05:20:49.720049 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.100.62:35003: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":124,"threads_started":1}
I20260812 05:20:49.720752 11886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:49.720558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.62:35003 (local address 127.31.100.3:36911)
0812 05:20:49.720675 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0812 05:20:49.720709 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.100.62:35003: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":66,"threads_started":1}
I20260812 05:20:50.720821 11888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:50.720603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.62:35003 (local address 127.31.100.1:47869)
0812 05:20:50.720727 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0812 05:20:50.720763 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.100.62:35003: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":62,"threads_started":1}
I20260812 05:20:50.720942 11889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:50.720654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.62:35003 (local address 127.31.100.2:54175)
0812 05:20:50.720761 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0812 05:20:50.720875 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.100.62:35003: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89,"thread_start_us":52,"threads_started":1}
I20260812 05:20:50.721385 11893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:50.721147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.62:35003 (local address 127.31.100.3:57981)
0812 05:20:50.721306 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0812 05:20:50.721342 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.100.62:35003: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":48,"threads_started":1}
I20260812 05:20:50.959487 11895 raft_consensus.cc:482] T 00000000000000000000000000000000 P 22c6383cda19494f95cfee62ca4d0ccf [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 070ed78c702d4333919d5d31da7b7608)
I20260812 05:20:50.959604 11895 raft_consensus.cc:504] T 00000000000000000000000000000000 P 22c6383cda19494f95cfee62ca4d0ccf [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "070ed78c702d4333919d5d31da7b7608" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 35003 } } peers { permanent_uuid: "e8be870d7089424d82f5be0171ba5242" member_type: VOTER last_known_addr { host: "127.31.100.61" port: 34209 } attrs { promote: false } } peers { permanent_uuid: "22c6383cda19494f95cfee62ca4d0ccf" member_type: VOTER last_known_addr { host: "127.31.100.60" port: 37129 } attrs { promote: false } }
I20260812 05:20:50.959777 11895 leader_election.cc:288] T 00000000000000000000000000000000 P 22c6383cda19494f95cfee62ca4d0ccf [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 070ed78c702d4333919d5d31da7b7608 (127.31.100.62:35003), e8be870d7089424d82f5be0171ba5242 (127.31.100.61:34209)
I20260812 05:20:50.960181 11896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:50.959927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.62:35003 (local address 127.0.0.1:34280)
0812 05:20:50.960080 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0812 05:20:50.960133 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.100.62:35003: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":71,"threads_started":1}
I20260812 05:20:50.960240 10715 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "22c6383cda19494f95cfee62ca4d0ccf" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "e8be870d7089424d82f5be0171ba5242" is_pre_election: true
I20260812 05:20:50.960379 10715 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e8be870d7089424d82f5be0171ba5242 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22c6383cda19494f95cfee62ca4d0ccf in term 1.
I20260812 05:20:50.960573 11416 leader_election.cc:302] T 00000000000000000000000000000000 P 22c6383cda19494f95cfee62ca4d0ccf [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22c6383cda19494f95cfee62ca4d0ccf, e8be870d7089424d82f5be0171ba5242; no voters:
I20260812 05:20:50.960625 11896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:50.960509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.62:35003 (local address 127.0.0.1:34290)
0812 05:20:50.960556 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0812 05:20:50.960586 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.100.62:35003: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260812 05:20:50.960664 11416 leader_election.cc:334] T 00000000000000000000000000000000 P 22c6383cda19494f95cfee62ca4d0ccf [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 070ed78c702d4333919d5d31da7b7608 (127.31.100.62:35003): Network error: Client connection negotiation failed: client connection to 127.31.100.62:35003: connect: Connection refused (error 111)
I20260812 05:20:50.960693 11895 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 22c6383cda19494f95cfee62ca4d0ccf [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260812 05:20:50.960743 11895 raft_consensus.cc:482] T 00000000000000000000000000000000 P 22c6383cda19494f95cfee62ca4d0ccf [term 1 FOLLOWER]: Starting leader election (detected failure of leader 070ed78c702d4333919d5d31da7b7608)
I20260812 05:20:50.960767 11895 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 22c6383cda19494f95cfee62ca4d0ccf [term 1 FOLLOWER]: Advancing to term 2
I20260812 05:20:50.961504 11895 raft_consensus.cc:504] T 00000000000000000000000000000000 P 22c6383cda19494f95cfee62ca4d0ccf [term 2 FOLLOWER]: Starting leader election with config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "070ed78c702d4333919d5d31da7b7608" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 35003 } } peers { permanent_uuid: "e8be870d7089424d82f5be0171ba5242" member_type: VOTER last_known_addr { host: "127.31.100.61" port: 34209 } attrs { promote: false } } peers { permanent_uuid: "22c6383cda19494f95cfee62ca4d0ccf" member_type: VOTER last_known_addr { host: "127.31.100.60" port: 37129 } attrs { promote: false } }
I20260812 05:20:50.961624 11895 leader_election.cc:288] T 00000000000000000000000000000000 P 22c6383cda19494f95cfee62ca4d0ccf [CANDIDATE]: Term 2 election: Requested vote from peers 070ed78c702d4333919d5d31da7b7608 (127.31.100.62:35003), e8be870d7089424d82f5be0171ba5242 (127.31.100.61:34209)
I20260812 05:20:50.961805 11896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:50.961716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.62:35003 (local address 127.0.0.1:34292)
0812 05:20:50.961751 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0812 05:20:50.961778 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.100.62:35003: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":16}
I20260812 05:20:50.961861 10715 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "22c6383cda19494f95cfee62ca4d0ccf" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "e8be870d7089424d82f5be0171ba5242"
I20260812 05:20:50.961939 10715 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e8be870d7089424d82f5be0171ba5242 [term 1 FOLLOWER]: Advancing to term 2
I20260812 05:20:50.962100 11896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:50.962037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.62:35003 (local address 127.0.0.1:34294)
0812 05:20:50.962064 (+ 27us) negotiation.cc:107] Waiting for socket to connect
0812 05:20:50.962087 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.100.62:35003: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":16}
W20260812 05:20:50.962165 11416 leader_election.cc:334] T 00000000000000000000000000000000 P 22c6383cda19494f95cfee62ca4d0ccf [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 070ed78c702d4333919d5d31da7b7608 (127.31.100.62:35003): Network error: Client connection negotiation failed: client connection to 127.31.100.62:35003: connect: Connection refused (error 111)
I20260812 05:20:50.962735 10715 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e8be870d7089424d82f5be0171ba5242 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22c6383cda19494f95cfee62ca4d0ccf in term 2.
I20260812 05:20:50.962877 11416 leader_election.cc:302] T 00000000000000000000000000000000 P 22c6383cda19494f95cfee62ca4d0ccf [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 22c6383cda19494f95cfee62ca4d0ccf, e8be870d7089424d82f5be0171ba5242; no voters: 070ed78c702d4333919d5d31da7b7608
I20260812 05:20:50.962977 11895 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 22c6383cda19494f95cfee62ca4d0ccf [term 2 FOLLOWER]: Leader election won for term 2
I20260812 05:20:50.963084 11895 raft_consensus.cc:686] T 00000000000000000000000000000000 P 22c6383cda19494f95cfee62ca4d0ccf [term 2 LEADER]: Becoming Leader. State: Replica: 22c6383cda19494f95cfee62ca4d0ccf, State: Running, Role: LEADER
I20260812 05:20:50.963160 11895 consensus_queue.cc:237] T 00000000000000000000000000000000 P 22c6383cda19494f95cfee62ca4d0ccf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "070ed78c702d4333919d5d31da7b7608" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 35003 } } peers { permanent_uuid: "e8be870d7089424d82f5be0171ba5242" member_type: VOTER last_known_addr { host: "127.31.100.61" port: 34209 } attrs { promote: false } } peers { permanent_uuid: "22c6383cda19494f95cfee62ca4d0ccf" member_type: VOTER last_known_addr { host: "127.31.100.60" port: 37129 } attrs { promote: false } }
I20260812 05:20:50.963485 11902 sys_catalog.cc:455] T 00000000000000000000000000000000 P 22c6383cda19494f95cfee62ca4d0ccf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 22c6383cda19494f95cfee62ca4d0ccf. Latest consensus state: current_term: 2 leader_uuid: "22c6383cda19494f95cfee62ca4d0ccf" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "070ed78c702d4333919d5d31da7b7608" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 35003 } } peers { permanent_uuid: "e8be870d7089424d82f5be0171ba5242" member_type: VOTER last_known_addr { host: "127.31.100.61" port: 34209 } attrs { promote: false } } peers { permanent_uuid: "22c6383cda19494f95cfee62ca4d0ccf" member_type: NON_VOTER last_known_addr { host: "127.31.100.60" port: 37129 } attrs { promote: true } } } pending_config { opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "070ed78c702d4333919d5d31da7b7608" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 35003 } } peers { permanent_uuid: "e8be870d7089424d82f5be0171ba5242" member_type: VOTER last_known_addr { host: "127.31.100.61" port: 34209 } attrs { promote: false } } peers { permanent_uuid: "22c6383cda19494f95cfee62ca4d0ccf" member_type: VOTER last_known_addr { host: "127.31.100.60" port: 37129 } attrs { promote: false } } }
I20260812 05:20:50.963568 11902 sys_catalog.cc:458] T 00000000000000000000000000000000 P 22c6383cda19494f95cfee62ca4d0ccf [sys.catalog]: This master's current role is: LEADER
I20260812 05:20:50.963742 11904 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 05:20:50.964138 11904 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 05:20:50.964207 11904 catalog_manager.cc:1259] Loaded cluster ID: 1c12969b4cd344eab17b8b9fb1e66566
I20260812 05:20:50.964237 11904 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 05:20:50.964409 11904 catalog_manager.cc:1504] Loading token signing keys...
I20260812 05:20:50.964488 11904 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 22c6383cda19494f95cfee62ca4d0ccf: Loaded TSK: 0
I20260812 05:20:50.964627 11904 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 05:20:50.983130 11440 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55782:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 05:20:50.984709 11440 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 05:20:50.985704 11896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:50.985608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.62:35003 (local address 127.0.0.1:34308)
0812 05:20:50.985652 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0812 05:20:50.985680 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.100.62:35003: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260812 05:20:50.985692 10715 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e8be870d7089424d82f5be0171ba5242 [term 2 FOLLOWER]: Refusing update from remote peer 22c6383cda19494f95cfee62ca4d0ccf: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260812 05:20:50.986025 11896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:50.985937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.62:35003 (local address 127.0.0.1:34316)
0812 05:20:50.985975 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0812 05:20:50.986002 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.100.62:35003: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
I20260812 05:20:50.986034 11902 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 22c6383cda19494f95cfee62ca4d0ccf [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8be870d7089424d82f5be0171ba5242" member_type: VOTER last_known_addr { host: "127.31.100.61" port: 34209 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.000s
W20260812 05:20:50.986162 11416 consensus_peers.cc:487] T 00000000000000000000000000000000 P 22c6383cda19494f95cfee62ca4d0ccf -> Peer 070ed78c702d4333919d5d31da7b7608 (127.31.100.62:35003): Couldn't send request to peer 070ed78c702d4333919d5d31da7b7608. Status: Network error: Client connection negotiation failed: client connection to 127.31.100.62:35003: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260812 05:20:50.986721 11906 sys_catalog.cc:455] T 00000000000000000000000000000000 P e8be870d7089424d82f5be0171ba5242 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 22c6383cda19494f95cfee62ca4d0ccf. Latest consensus state: current_term: 2 leader_uuid: "22c6383cda19494f95cfee62ca4d0ccf" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "070ed78c702d4333919d5d31da7b7608" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 35003 } } peers { permanent_uuid: "e8be870d7089424d82f5be0171ba5242" member_type: VOTER last_known_addr { host: "127.31.100.61" port: 34209 } attrs { promote: false } } peers { permanent_uuid: "22c6383cda19494f95cfee62ca4d0ccf" member_type: NON_VOTER last_known_addr { host: "127.31.100.60" port: 37129 } attrs { promote: true } } } pending_config { opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "070ed78c702d4333919d5d31da7b7608" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 35003 } } peers { permanent_uuid: "e8be870d7089424d82f5be0171ba5242" member_type: VOTER last_known_addr { host: "127.31.100.61" port: 34209 } attrs { promote: false } } peers { permanent_uuid: "22c6383cda19494f95cfee62ca4d0ccf" member_type: VOTER last_known_addr { host: "127.31.100.60" port: 37129 } attrs { promote: false } } }
I20260812 05:20:50.986837 11906 sys_catalog.cc:458] T 00000000000000000000000000000000 P e8be870d7089424d82f5be0171ba5242 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 05:20:50.987144 11895 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 22c6383cda19494f95cfee62ca4d0ccf [term 2 LEADER]: Committing config change with OpId 1.12: config changed from index 7 to 12, 22c6383cda19494f95cfee62ca4d0ccf (127.31.100.60) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "070ed78c702d4333919d5d31da7b7608" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 35003 } } peers { permanent_uuid: "e8be870d7089424d82f5be0171ba5242" member_type: VOTER last_known_addr { host: "127.31.100.61" port: 34209 } attrs { promote: false } } peers { permanent_uuid: "22c6383cda19494f95cfee62ca4d0ccf" member_type: VOTER last_known_addr { host: "127.31.100.60" port: 37129 } attrs { promote: false } } }
I20260812 05:20:50.987346 10715 raft_consensus.cc:2932] T 00000000000000000000000000000000 P e8be870d7089424d82f5be0171ba5242 [term 2 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 7 to 12, 22c6383cda19494f95cfee62ca4d0ccf (127.31.100.60) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "070ed78c702d4333919d5d31da7b7608" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 35003 } } peers { permanent_uuid: "e8be870d7089424d82f5be0171ba5242" member_type: VOTER last_known_addr { host: "127.31.100.61" port: 34209 } attrs { promote: false } } peers { permanent_uuid: "22c6383cda19494f95cfee62ca4d0ccf" member_type: VOTER last_known_addr { host: "127.31.100.60" port: 37129 } attrs { promote: false } } }
I20260812 05:20:50.988369 11905 mvcc.cc:205] Tried to move back new op lower bound from 7317553360835547136 to 7317553360745496576. Current Snapshot: MvccSnapshot[applied={T|T < 7317553360835547136}]
I20260812 05:20:50.988335 11902 sys_catalog.cc:455] T 00000000000000000000000000000000 P 22c6383cda19494f95cfee62ca4d0ccf [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "22c6383cda19494f95cfee62ca4d0ccf" committed_config { opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "070ed78c702d4333919d5d31da7b7608" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 35003 } } peers { permanent_uuid: "e8be870d7089424d82f5be0171ba5242" member_type: VOTER last_known_addr { host: "127.31.100.61" port: 34209 } attrs { promote: false } } peers { permanent_uuid: "22c6383cda19494f95cfee62ca4d0ccf" member_type: VOTER last_known_addr { host: "127.31.100.60" port: 37129 } attrs { promote: false } } }
I20260812 05:20:50.988338 11895 sys_catalog.cc:455] T 00000000000000000000000000000000 P 22c6383cda19494f95cfee62ca4d0ccf [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "22c6383cda19494f95cfee62ca4d0ccf" committed_config { opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "070ed78c702d4333919d5d31da7b7608" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 35003 } } peers { permanent_uuid: "e8be870d7089424d82f5be0171ba5242" member_type: VOTER last_known_addr { host: "127.31.100.61" port: 34209 } attrs { promote: false } } peers { permanent_uuid: "22c6383cda19494f95cfee62ca4d0ccf" member_type: VOTER last_known_addr { host: "127.31.100.60" port: 37129 } attrs { promote: false } } }
I20260812 05:20:50.988538 11902 sys_catalog.cc:458] T 00000000000000000000000000000000 P 22c6383cda19494f95cfee62ca4d0ccf [sys.catalog]: This master's current role is: LEADER
I20260812 05:20:50.988551 11895 sys_catalog.cc:458] T 00000000000000000000000000000000 P 22c6383cda19494f95cfee62ca4d0ccf [sys.catalog]: This master's current role is: LEADER
I20260812 05:20:50.988409 11909 sys_catalog.cc:455] T 00000000000000000000000000000000 P e8be870d7089424d82f5be0171ba5242 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "22c6383cda19494f95cfee62ca4d0ccf" committed_config { opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "070ed78c702d4333919d5d31da7b7608" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 35003 } } peers { permanent_uuid: "e8be870d7089424d82f5be0171ba5242" member_type: VOTER last_known_addr { host: "127.31.100.61" port: 34209 } attrs { promote: false } } peers { permanent_uuid: "22c6383cda19494f95cfee62ca4d0ccf" member_type: VOTER last_known_addr { host: "127.31.100.60" port: 37129 } attrs { promote: false } } }
I20260812 05:20:50.988524 11906 sys_catalog.cc:455] T 00000000000000000000000000000000 P e8be870d7089424d82f5be0171ba5242 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "22c6383cda19494f95cfee62ca4d0ccf" committed_config { opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "070ed78c702d4333919d5d31da7b7608" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 35003 } } peers { permanent_uuid: "e8be870d7089424d82f5be0171ba5242" member_type: VOTER last_known_addr { host: "127.31.100.61" port: 34209 } attrs { promote: false } } peers { permanent_uuid: "22c6383cda19494f95cfee62ca4d0ccf" member_type: VOTER last_known_addr { host: "127.31.100.60" port: 37129 } attrs { promote: false } } }
I20260812 05:20:50.988653 11907 mvcc.cc:205] Tried to move back new op lower bound from 7317553360835547136 to 7317553360745496576. Current Snapshot: MvccSnapshot[applied={T|T < 7317553360835547136}]
I20260812 05:20:50.988749 11909 sys_catalog.cc:458] T 00000000000000000000000000000000 P e8be870d7089424d82f5be0171ba5242 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 05:20:50.988763 11906 sys_catalog.cc:458] T 00000000000000000000000000000000 P e8be870d7089424d82f5be0171ba5242 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 05:20:50.993794 11918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:50.990587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55022 (local address 127.31.100.2:36081)
0812 05:20:50.990922 (+ 335us) server_negotiation.cc:206] Beginning negotiation
0812 05:20:50.990924 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 05:20:50.990932 (+ 8us) server_negotiation.cc:407] Connection header received
0812 05:20:50.990957 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:20:50.990958 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:20:50.990968 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:20:50.991031 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 05:20:50.991198 (+ 167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:50.991831 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:50.993011 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:50.993410 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:50.993516 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:50.993530 (+ 14us) server_negotiation.cc:1035] Waiting for connection context
0812 05:20:50.993675 (+ 145us) server_negotiation.cc:299] Negotiation successful
0812 05:20:50.993727 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":314,"thread_start_us":48,"threads_started":1}
I20260812 05:20:50.993796 11896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:50.990381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.1:44127 (local address 127.0.0.1:59728)
0812 05:20:50.990487 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0812 05:20:50.990497 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0812 05:20:50.990547 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:20:50.990713 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:20:50.990714 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:20:50.990715 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 05:20:50.990784 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:20:50.990786 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:20:50.991684 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:20:50.991685 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:20:50.992805 (+ 1120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:20:50.992807 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:20:50.993598 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:20:50.993599 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:20:50.993666 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:50.993669 (+ 3us) client_negotiation.cc:769] Sending connection context
0812 05:20:50.993713 (+ 44us) client_negotiation.cc:240] Negotiation successful
0812 05:20:50.993746 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":91}
I20260812 05:20:50.993788 11914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:50.990397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.2:36081 (local address 127.0.0.1:55022)
0812 05:20:50.990796 (+ 399us) negotiation.cc:107] Waiting for socket to connect
0812 05:20:50.990805 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0812 05:20:50.990850 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:20:50.991074 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:20:50.991075 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:20:50.991077 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 05:20:50.991137 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:20:50.991138 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:20:50.991897 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:20:50.991898 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:20:50.992941 (+ 1043us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:20:50.992944 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:20:50.993517 (+ 573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:20:50.993518 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:20:50.993612 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:50.993617 (+ 5us) client_negotiation.cc:769] Sending connection context
0812 05:20:50.993669 (+ 52us) client_negotiation.cc:240] Negotiation successful
0812 05:20:50.993707 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":379,"thread_start_us":88,"threads_started":1}
I20260812 05:20:50.993947 11915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:50.990454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59728 (local address 127.31.100.1:44127)
0812 05:20:50.990584 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0812 05:20:50.990586 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 05:20:50.990611 (+ 25us) server_negotiation.cc:407] Connection header received
0812 05:20:50.990637 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:20:50.990639 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:20:50.990650 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:20:50.990700 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 05:20:50.990834 (+ 134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:50.991577 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:50.992899 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:50.993498 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:50.993594 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:50.993611 (+ 17us) server_negotiation.cc:1035] Waiting for connection context
0812 05:20:50.993827 (+ 216us) server_negotiation.cc:299] Negotiation successful
0812 05:20:50.993879 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107,"thread_start_us":68,"threads_started":1}
I20260812 05:20:50.994164 11043 tablet_service.cc:1467] Processing CreateTablet for tablet 767023f6d890463f929727a00665d802 (DEFAULT_TABLE table=test-table [id=4f62e579d7a84a7baeb20b0e5e44c7cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 05:20:50.994273 11043 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 767023f6d890463f929727a00665d802. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 05:20:50.994642 11917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:50.990480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.3:39085 (local address 127.0.0.1:39158)
0812 05:20:50.990919 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0812 05:20:50.990937 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0812 05:20:50.990983 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:20:50.991101 (+ 118us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:20:50.991101 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:20:50.991103 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 05:20:50.991136 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:20:50.991138 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:20:50.992453 (+ 1315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:20:50.992454 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:20:50.993568 (+ 1114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:20:50.993571 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:20:50.994399 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:20:50.994401 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:20:50.994474 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:50.994478 (+ 4us) client_negotiation.cc:769] Sending connection context
0812 05:20:50.994525 (+ 47us) client_negotiation.cc:240] Negotiation successful
0812 05:20:50.994564 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":424,"thread_start_us":74,"threads_started":1}
I20260812 05:20:50.994649 11920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:50.990685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39158 (local address 127.31.100.3:39085)
0812 05:20:50.990793 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0812 05:20:50.990795 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 05:20:50.990990 (+ 195us) server_negotiation.cc:407] Connection header received
0812 05:20:50.991028 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:20:50.991029 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:20:50.991040 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:20:50.991102 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 05:20:50.991913 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:50.992373 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:50.993643 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:50.994207 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:50.994276 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:50.994287 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0812 05:20:50.994539 (+ 252us) server_negotiation.cc:299] Negotiation successful
0812 05:20:50.994583 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87,"thread_start_us":59,"threads_started":1}
I20260812 05:20:50.994971 11275 tablet_service.cc:1467] Processing CreateTablet for tablet 767023f6d890463f929727a00665d802 (DEFAULT_TABLE table=test-table [id=4f62e579d7a84a7baeb20b0e5e44c7cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 05:20:50.995096 11275 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 767023f6d890463f929727a00665d802. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 05:20:50.995507 11922 tablet_bootstrap.cc:492] T 767023f6d890463f929727a00665d802 P f54f92d5624f42a9b11d653eb7bd017c: Bootstrap starting.
I20260812 05:20:50.995827 11922 tablet_bootstrap.cc:654] T 767023f6d890463f929727a00665d802 P f54f92d5624f42a9b11d653eb7bd017c: Neither blocks nor log segments found. Creating new log.
I20260812 05:20:50.996244 11925 tablet_bootstrap.cc:492] T 767023f6d890463f929727a00665d802 P bf52456a845341ebbf4e4e7405844a2d: Bootstrap starting.
I20260812 05:20:50.996304 11922 tablet_bootstrap.cc:492] T 767023f6d890463f929727a00665d802 P f54f92d5624f42a9b11d653eb7bd017c: No bootstrap required, opened a new log
I20260812 05:20:50.996429 11922 ts_tablet_manager.cc:1397] T 767023f6d890463f929727a00665d802 P f54f92d5624f42a9b11d653eb7bd017c: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260812 05:20:50.996465 11630 tablet_service.cc:1467] Processing CreateTablet for tablet 767023f6d890463f929727a00665d802 (DEFAULT_TABLE table=test-table [id=4f62e579d7a84a7baeb20b0e5e44c7cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 05:20:50.996585 11630 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 767023f6d890463f929727a00665d802. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 05:20:50.996555 11922 raft_consensus.cc:348] T 767023f6d890463f929727a00665d802 P f54f92d5624f42a9b11d653eb7bd017c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54f92d5624f42a9b11d653eb7bd017c" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 44127 } } peers { permanent_uuid: "bf52456a845341ebbf4e4e7405844a2d" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 36081 } } peers { permanent_uuid: "167b45749e0d4747866e5be16306875f" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 39085 } }
I20260812 05:20:50.996603 11925 tablet_bootstrap.cc:654] T 767023f6d890463f929727a00665d802 P bf52456a845341ebbf4e4e7405844a2d: Neither blocks nor log segments found. Creating new log.
I20260812 05:20:50.996619 11922 raft_consensus.cc:374] T 767023f6d890463f929727a00665d802 P f54f92d5624f42a9b11d653eb7bd017c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 05:20:50.996647 11922 raft_consensus.cc:729] T 767023f6d890463f929727a00665d802 P f54f92d5624f42a9b11d653eb7bd017c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f54f92d5624f42a9b11d653eb7bd017c, State: Initialized, Role: FOLLOWER
I20260812 05:20:50.996707 11922 consensus_queue.cc:260] T 767023f6d890463f929727a00665d802 P f54f92d5624f42a9b11d653eb7bd017c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54f92d5624f42a9b11d653eb7bd017c" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 44127 } } peers { permanent_uuid: "bf52456a845341ebbf4e4e7405844a2d" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 36081 } } peers { permanent_uuid: "167b45749e0d4747866e5be16306875f" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 39085 } }
I20260812 05:20:50.996874 11922 ts_tablet_manager.cc:1428] T 767023f6d890463f929727a00665d802 P f54f92d5624f42a9b11d653eb7bd017c: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260812 05:20:50.997188 11925 tablet_bootstrap.cc:492] T 767023f6d890463f929727a00665d802 P bf52456a845341ebbf4e4e7405844a2d: No bootstrap required, opened a new log
I20260812 05:20:50.997239 11925 ts_tablet_manager.cc:1397] T 767023f6d890463f929727a00665d802 P bf52456a845341ebbf4e4e7405844a2d: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260812 05:20:50.997357 11925 raft_consensus.cc:348] T 767023f6d890463f929727a00665d802 P bf52456a845341ebbf4e4e7405844a2d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54f92d5624f42a9b11d653eb7bd017c" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 44127 } } peers { permanent_uuid: "bf52456a845341ebbf4e4e7405844a2d" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 36081 } } peers { permanent_uuid: "167b45749e0d4747866e5be16306875f" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 39085 } }
I20260812 05:20:50.997421 11925 raft_consensus.cc:374] T 767023f6d890463f929727a00665d802 P bf52456a845341ebbf4e4e7405844a2d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 05:20:50.997464 11925 raft_consensus.cc:729] T 767023f6d890463f929727a00665d802 P bf52456a845341ebbf4e4e7405844a2d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf52456a845341ebbf4e4e7405844a2d, State: Initialized, Role: FOLLOWER
I20260812 05:20:50.997543 11151 heartbeater.cc:503] Master 127.31.100.60:37129 was elected leader, sending a full tablet report...
I20260812 05:20:50.997524 11925 consensus_queue.cc:260] T 767023f6d890463f929727a00665d802 P bf52456a845341ebbf4e4e7405844a2d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54f92d5624f42a9b11d653eb7bd017c" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 44127 } } peers { permanent_uuid: "bf52456a845341ebbf4e4e7405844a2d" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 36081 } } peers { permanent_uuid: "167b45749e0d4747866e5be16306875f" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 39085 } }
I20260812 05:20:50.998129 11937 tablet_bootstrap.cc:492] T 767023f6d890463f929727a00665d802 P 167b45749e0d4747866e5be16306875f: Bootstrap starting.
I20260812 05:20:50.998198 11925 ts_tablet_manager.cc:1428] T 767023f6d890463f929727a00665d802 P bf52456a845341ebbf4e4e7405844a2d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260812 05:20:50.998312 11888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:50.998220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.62:35003 (local address 127.31.100.1:34415)
0812 05:20:50.998258 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0812 05:20:50.998287 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.100.62:35003: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":20}
I20260812 05:20:50.998653 11889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:50.998554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.62:35003 (local address 127.31.100.2:49109)
0812 05:20:50.998590 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0812 05:20:50.998627 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.100.62:35003: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":19}
I20260812 05:20:50.998654 11382 heartbeater.cc:503] Master 127.31.100.60:37129 was elected leader, sending a full tablet report...
I20260812 05:20:50.998698 11937 tablet_bootstrap.cc:654] T 767023f6d890463f929727a00665d802 P 167b45749e0d4747866e5be16306875f: Neither blocks nor log segments found. Creating new log.
I20260812 05:20:50.999074 11889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:50.998951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.62:35003 (local address 127.31.100.2:60483)
0812 05:20:50.999019 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0812 05:20:50.999049 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.100.62:35003: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260812 05:20:50.999449 11937 tablet_bootstrap.cc:492] T 767023f6d890463f929727a00665d802 P 167b45749e0d4747866e5be16306875f: No bootstrap required, opened a new log
I20260812 05:20:50.999496 11937 ts_tablet_manager.cc:1397] T 767023f6d890463f929727a00665d802 P 167b45749e0d4747866e5be16306875f: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260812 05:20:50.999615 11937 raft_consensus.cc:348] T 767023f6d890463f929727a00665d802 P 167b45749e0d4747866e5be16306875f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54f92d5624f42a9b11d653eb7bd017c" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 44127 } } peers { permanent_uuid: "bf52456a845341ebbf4e4e7405844a2d" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 36081 } } peers { permanent_uuid: "167b45749e0d4747866e5be16306875f" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 39085 } }
I20260812 05:20:50.999673 11937 raft_consensus.cc:374] T 767023f6d890463f929727a00665d802 P 167b45749e0d4747866e5be16306875f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 05:20:50.999686 11937 raft_consensus.cc:729] T 767023f6d890463f929727a00665d802 P 167b45749e0d4747866e5be16306875f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 167b45749e0d4747866e5be16306875f, State: Initialized, Role: FOLLOWER
I20260812 05:20:50.999727 11937 consensus_queue.cc:260] T 767023f6d890463f929727a00665d802 P 167b45749e0d4747866e5be16306875f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54f92d5624f42a9b11d653eb7bd017c" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 44127 } } peers { permanent_uuid: "bf52456a845341ebbf4e4e7405844a2d" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 36081 } } peers { permanent_uuid: "167b45749e0d4747866e5be16306875f" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 39085 } }
I20260812 05:20:50.999910 11937 ts_tablet_manager.cc:1428] T 767023f6d890463f929727a00665d802 P 167b45749e0d4747866e5be16306875f: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260812 05:20:51.000207 11893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:51.000111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.62:35003 (local address 127.31.100.3:44391)
0812 05:20:51.000147 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0812 05:20:51.000180 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.100.62:35003: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":19}
I20260812 05:20:51.000296 11750 heartbeater.cc:503] Master 127.31.100.60:37129 was elected leader, sending a full tablet report...
W20260812 05:20:51.006633 11755 tablet.cc:2369] T 767023f6d890463f929727a00665d802 P 167b45749e0d4747866e5be16306875f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 05:20:51.017931 11927 raft_consensus.cc:482] T 767023f6d890463f929727a00665d802 P f54f92d5624f42a9b11d653eb7bd017c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 05:20:51.018011 11927 raft_consensus.cc:504] T 767023f6d890463f929727a00665d802 P f54f92d5624f42a9b11d653eb7bd017c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54f92d5624f42a9b11d653eb7bd017c" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 44127 } } peers { permanent_uuid: "bf52456a845341ebbf4e4e7405844a2d" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 36081 } } peers { permanent_uuid: "167b45749e0d4747866e5be16306875f" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 39085 } }
I20260812 05:20:51.018178 11927 leader_election.cc:288] T 767023f6d890463f929727a00665d802 P f54f92d5624f42a9b11d653eb7bd017c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bf52456a845341ebbf4e4e7405844a2d (127.31.100.2:36081), 167b45749e0d4747866e5be16306875f (127.31.100.3:39085)
I20260812 05:20:51.018380 11307 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "767023f6d890463f929727a00665d802" candidate_uuid: "f54f92d5624f42a9b11d653eb7bd017c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf52456a845341ebbf4e4e7405844a2d" is_pre_election: true
I20260812 05:20:51.018399 11673 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "767023f6d890463f929727a00665d802" candidate_uuid: "f54f92d5624f42a9b11d653eb7bd017c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "167b45749e0d4747866e5be16306875f" is_pre_election: true
I20260812 05:20:51.018477 11673 raft_consensus.cc:2454] T 767023f6d890463f929727a00665d802 P 167b45749e0d4747866e5be16306875f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f54f92d5624f42a9b11d653eb7bd017c in term 0.
I20260812 05:20:51.018476 11307 raft_consensus.cc:2454] T 767023f6d890463f929727a00665d802 P bf52456a845341ebbf4e4e7405844a2d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f54f92d5624f42a9b11d653eb7bd017c in term 0.
I20260812 05:20:51.018615 10975 leader_election.cc:302] T 767023f6d890463f929727a00665d802 P f54f92d5624f42a9b11d653eb7bd017c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 167b45749e0d4747866e5be16306875f, f54f92d5624f42a9b11d653eb7bd017c; no voters:
I20260812 05:20:51.018738 11927 raft_consensus.cc:2781] T 767023f6d890463f929727a00665d802 P f54f92d5624f42a9b11d653eb7bd017c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 05:20:51.018783 11927 raft_consensus.cc:482] T 767023f6d890463f929727a00665d802 P f54f92d5624f42a9b11d653eb7bd017c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 05:20:51.018796 11927 raft_consensus.cc:3037] T 767023f6d890463f929727a00665d802 P f54f92d5624f42a9b11d653eb7bd017c [term 0 FOLLOWER]: Advancing to term 1
I20260812 05:20:51.019487 11927 raft_consensus.cc:504] T 767023f6d890463f929727a00665d802 P f54f92d5624f42a9b11d653eb7bd017c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54f92d5624f42a9b11d653eb7bd017c" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 44127 } } peers { permanent_uuid: "bf52456a845341ebbf4e4e7405844a2d" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 36081 } } peers { permanent_uuid: "167b45749e0d4747866e5be16306875f" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 39085 } }
I20260812 05:20:51.019603 11927 leader_election.cc:288] T 767023f6d890463f929727a00665d802 P f54f92d5624f42a9b11d653eb7bd017c [CANDIDATE]: Term 1 election: Requested vote from peers bf52456a845341ebbf4e4e7405844a2d (127.31.100.2:36081), 167b45749e0d4747866e5be16306875f (127.31.100.3:39085)
I20260812 05:20:51.019759 11307 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "767023f6d890463f929727a00665d802" candidate_uuid: "f54f92d5624f42a9b11d653eb7bd017c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf52456a845341ebbf4e4e7405844a2d"
I20260812 05:20:51.019827 11307 raft_consensus.cc:3037] T 767023f6d890463f929727a00665d802 P bf52456a845341ebbf4e4e7405844a2d [term 0 FOLLOWER]: Advancing to term 1
I20260812 05:20:51.019802 11673 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "767023f6d890463f929727a00665d802" candidate_uuid: "f54f92d5624f42a9b11d653eb7bd017c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "167b45749e0d4747866e5be16306875f"
I20260812 05:20:51.019879 11673 raft_consensus.cc:3037] T 767023f6d890463f929727a00665d802 P 167b45749e0d4747866e5be16306875f [term 0 FOLLOWER]: Advancing to term 1
I20260812 05:20:51.020449 11673 raft_consensus.cc:2454] T 767023f6d890463f929727a00665d802 P 167b45749e0d4747866e5be16306875f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f54f92d5624f42a9b11d653eb7bd017c in term 1.
I20260812 05:20:51.020459 11307 raft_consensus.cc:2454] T 767023f6d890463f929727a00665d802 P bf52456a845341ebbf4e4e7405844a2d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f54f92d5624f42a9b11d653eb7bd017c in term 1.
I20260812 05:20:51.020613 10975 leader_election.cc:302] T 767023f6d890463f929727a00665d802 P f54f92d5624f42a9b11d653eb7bd017c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 167b45749e0d4747866e5be16306875f, f54f92d5624f42a9b11d653eb7bd017c; no voters:
I20260812 05:20:51.020713 11927 raft_consensus.cc:2781] T 767023f6d890463f929727a00665d802 P f54f92d5624f42a9b11d653eb7bd017c [term 1 FOLLOWER]: Leader election won for term 1
I20260812 05:20:51.020851 11927 raft_consensus.cc:686] T 767023f6d890463f929727a00665d802 P f54f92d5624f42a9b11d653eb7bd017c [term 1 LEADER]: Becoming Leader. State: Replica: f54f92d5624f42a9b11d653eb7bd017c, State: Running, Role: LEADER
I20260812 05:20:51.020915 11927 consensus_queue.cc:237] T 767023f6d890463f929727a00665d802 P f54f92d5624f42a9b11d653eb7bd017c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54f92d5624f42a9b11d653eb7bd017c" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 44127 } } peers { permanent_uuid: "bf52456a845341ebbf4e4e7405844a2d" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 36081 } } peers { permanent_uuid: "167b45749e0d4747866e5be16306875f" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 39085 } }
I20260812 05:20:51.021215 11888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:51.021101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.62:35003 (local address 127.31.100.1:55795)
0812 05:20:51.021153 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0812 05:20:51.021183 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.100.62:35003: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":17}
I20260812 05:20:51.021508 11440 catalog_manager.cc:5495] T 767023f6d890463f929727a00665d802 P f54f92d5624f42a9b11d653eb7bd017c reported cstate change: term changed from 0 to 1, leader changed from <none> to f54f92d5624f42a9b11d653eb7bd017c (127.31.100.1). New cstate: current_term: 1 leader_uuid: "f54f92d5624f42a9b11d653eb7bd017c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54f92d5624f42a9b11d653eb7bd017c" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 44127 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bf52456a845341ebbf4e4e7405844a2d" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 36081 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "167b45749e0d4747866e5be16306875f" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 39085 } health_report { overall_health: UNKNOWN } } }
I20260812 05:20:51.028714 11915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:51.026640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59738 (local address 127.31.100.1:44127)
0812 05:20:51.026690 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0812 05:20:51.026692 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 05:20:51.026700 (+ 8us) server_negotiation.cc:407] Connection header received
0812 05:20:51.026748 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:20:51.026748 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:20:51.026757 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:20:51.026799 (+ 42us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 05:20:51.026905 (+ 106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:51.027378 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:51.028095 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:51.028287 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:51.028340 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:51.028510 (+ 170us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 05:20:51.028542 (+ 32us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 05:20:51.028595 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0812 05:20:51.028642 (+ 47us) server_negotiation.cc:299] Negotiation successful
0812 05:20:51.028663 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260812 05:20:51.029675 11673 raft_consensus.cc:1261] T 767023f6d890463f929727a00665d802 P 167b45749e0d4747866e5be16306875f [term 1 FOLLOWER]: Refusing update from remote peer f54f92d5624f42a9b11d653eb7bd017c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 05:20:51.029675 11307 raft_consensus.cc:1261] T 767023f6d890463f929727a00665d802 P bf52456a845341ebbf4e4e7405844a2d [term 1 FOLLOWER]: Refusing update from remote peer f54f92d5624f42a9b11d653eb7bd017c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 05:20:51.029922 11941 consensus_queue.cc:1035] T 767023f6d890463f929727a00665d802 P f54f92d5624f42a9b11d653eb7bd017c [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf52456a845341ebbf4e4e7405844a2d" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 36081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 05:20:51.030037 11927 consensus_queue.cc:1035] T 767023f6d890463f929727a00665d802 P f54f92d5624f42a9b11d653eb7bd017c [LEADER]: Connected to new peer: Peer: permanent_uuid: "167b45749e0d4747866e5be16306875f" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 39085 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 05:20:51.030813 11893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:51.030705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.62:35003 (local address 127.31.100.3:45309)
0812 05:20:51.030750 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0812 05:20:51.030781 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.100.62:35003: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260812 05:20:51.031002 11889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:51.030767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.62:35003 (local address 127.31.100.2:41003)
0812 05:20:51.030934 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0812 05:20:51.030963 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.100.62:35003: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":150}
I20260812 05:20:51.031585 11945 mvcc.cc:205] Tried to move back new op lower bound from 7317553361015529472 to 7317553360982224896. Current Snapshot: MvccSnapshot[applied={T|T < 7317553361015529472}]
I20260812 05:20:51.032002 11888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:51.031898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.62:35003 (local address 127.31.100.1:57749)
0812 05:20:51.031942 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0812 05:20:51.031975 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.100.62:35003: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":19}
I20260812 05:20:51.032395 11893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:51.032297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.62:35003 (local address 127.31.100.3:47069)
0812 05:20:51.032335 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0812 05:20:51.032368 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.100.62:35003: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
I20260812 05:20:51.032588 11888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:51.032275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.62:35003 (local address 127.31.100.1:52815)
0812 05:20:51.032531 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0812 05:20:51.032563 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.100.62:35003: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":238}
I20260812 05:20:51.033000 11889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:51.032885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.62:35003 (local address 127.31.100.2:41905)
0812 05:20:51.032942 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0812 05:20:51.032974 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.100.62:35003: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260812 05:20:51.033501 11441 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55782:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 05:20:51.033597 11441 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55782:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 05:20:51.034785 11441 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 22c6383cda19494f95cfee62ca4d0ccf: Sending DeleteTablet for 3 replicas of tablet 767023f6d890463f929727a00665d802
I20260812 05:20:51.035076 11630 tablet_service.cc:1514] Processing DeleteTablet for tablet 767023f6d890463f929727a00665d802 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 05:20:51 UTC) from {username='oryx', principal='oryx/127.31.100.60@KRBTEST.COM'} at 127.0.0.1:39158
I20260812 05:20:51.035143 11043 tablet_service.cc:1514] Processing DeleteTablet for tablet 767023f6d890463f929727a00665d802 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 05:20:51 UTC) from {username='oryx', principal='oryx/127.31.100.60@KRBTEST.COM'} at 127.0.0.1:59728
I20260812 05:20:51.035297 11275 tablet_service.cc:1514] Processing DeleteTablet for tablet 767023f6d890463f929727a00665d802 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 05:20:51 UTC) from {username='oryx', principal='oryx/127.31.100.60@KRBTEST.COM'} at 127.0.0.1:55022
I20260812 05:20:51.035321 11956 tablet_replica.cc:330] T 767023f6d890463f929727a00665d802 P 167b45749e0d4747866e5be16306875f: stopping tablet replica
I20260812 05:20:51.035372 11956 raft_consensus.cc:2229] T 767023f6d890463f929727a00665d802 P 167b45749e0d4747866e5be16306875f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 05:20:51.035414 11956 raft_consensus.cc:2258] T 767023f6d890463f929727a00665d802 P 167b45749e0d4747866e5be16306875f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 05:20:51.035485 11957 tablet_replica.cc:330] T 767023f6d890463f929727a00665d802 P f54f92d5624f42a9b11d653eb7bd017c: stopping tablet replica
I20260812 05:20:51.035522 11958 tablet_replica.cc:330] T 767023f6d890463f929727a00665d802 P bf52456a845341ebbf4e4e7405844a2d: stopping tablet replica
I20260812 05:20:51.035576 11958 raft_consensus.cc:2229] T 767023f6d890463f929727a00665d802 P bf52456a845341ebbf4e4e7405844a2d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 05:20:51.035624 11958 raft_consensus.cc:2258] T 767023f6d890463f929727a00665d802 P bf52456a845341ebbf4e4e7405844a2d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 05:20:51.035624 32144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0waFvT/build/release/bin/kudu with pid 10940
I20260812 05:20:51.044238 11956 ts_tablet_manager.cc:1905] T 767023f6d890463f929727a00665d802 P 167b45749e0d4747866e5be16306875f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 05:20:51.045722 11956 ts_tablet_manager.cc:1918] T 767023f6d890463f929727a00665d802 P 167b45749e0d4747866e5be16306875f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 05:20:51.045779 11956 log.cc:1196] T 767023f6d890463f929727a00665d802 P 167b45749e0d4747866e5be16306875f: Deleting WAL directory at /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786512017613366-32144-0/minicluster-data/ts-2/wal/wals/767023f6d890463f929727a00665d802
I20260812 05:20:51.045804 11958 ts_tablet_manager.cc:1905] T 767023f6d890463f929727a00665d802 P bf52456a845341ebbf4e4e7405844a2d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 05:20:51.046011 11956 ts_tablet_manager.cc:1939] T 767023f6d890463f929727a00665d802 P 167b45749e0d4747866e5be16306875f: Deleting consensus metadata
I20260812 05:20:51.046386 11420 catalog_manager.cc:4841] TS 167b45749e0d4747866e5be16306875f (127.31.100.3:39085): tablet 767023f6d890463f929727a00665d802 (table test-table [id=4f62e579d7a84a7baeb20b0e5e44c7cb]) successfully deleted
I20260812 05:20:51.047201 11958 ts_tablet_manager.cc:1918] T 767023f6d890463f929727a00665d802 P bf52456a845341ebbf4e4e7405844a2d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 05:20:51.047266 11958 log.cc:1196] T 767023f6d890463f929727a00665d802 P bf52456a845341ebbf4e4e7405844a2d: Deleting WAL directory at /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786512017613366-32144-0/minicluster-data/ts-1/wal/wals/767023f6d890463f929727a00665d802
I20260812 05:20:51.047465 11958 ts_tablet_manager.cc:1939] T 767023f6d890463f929727a00665d802 P bf52456a845341ebbf4e4e7405844a2d: Deleting consensus metadata
I20260812 05:20:51.047798 11416 catalog_manager.cc:4841] TS bf52456a845341ebbf4e4e7405844a2d (127.31.100.2:36081): tablet 767023f6d890463f929727a00665d802 (table test-table [id=4f62e579d7a84a7baeb20b0e5e44c7cb]) successfully deleted
I20260812 05:20:51.051607 11917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:51.051480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.1:44127 (local address 127.0.0.1:59744)
0812 05:20:51.051538 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0812 05:20:51.051571 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.100.1:44127: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260812 05:20:51.051772 32144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0waFvT/build/release/bin/kudu with pid 11160
W20260812 05:20:51.051810 11419 catalog_manager.cc:4584] TS f54f92d5624f42a9b11d653eb7bd017c (127.31.100.1:44127): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 767023f6d890463f929727a00665d802: Network error: Client connection negotiation failed: client connection to 127.31.100.1:44127: connect: Connection refused (error 111)
I20260812 05:20:51.060827 32144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0waFvT/build/release/bin/kudu with pid 11395
I20260812 05:20:51.072436 32144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0waFvT/build/release/bin/kudu with pid 10473
I20260812 05:20:51.081210 32144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0waFvT/build/release/bin/kudu with pid 10785
2026-08-12T05:20:51Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (7499 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786512017613366-32144-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 05:20:51 dist-test-slave-42z9 krb5kdc[11963](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 05:20:51 dist-test-slave-42z9 krb5kdc[11963](info): set up 2 sockets
Aug 12 05:20:51 dist-test-slave-42z9 krb5kdc[11963](info): commencing operation
krb5kdc: starting...
W20260812 05:20:52.249202 32144 mini_kdc.cc:121] Time spent starting KDC: real 1.140s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786512017613366-32144-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786512017613366-32144-0/krb5kdc/test-user.keytab.
Aug 12 05:20:52 dist-test-slave-42z9 krb5kdc[11963](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512052, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T05:20:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T05:20:52Z Disabled control of system clock
WARNING: no policy specified for oryx/127.31.100.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.31.100.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.31.100.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786512017613366-32144-0/krb5kdc/oryx_127.31.100.62.keytab.
Entry for principal oryx/127.31.100.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786512017613366-32144-0/krb5kdc/oryx_127.31.100.62.keytab.
I20260812 05:20:52.362386 32144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0waFvT/build/release/bin/kudu
/tmp/dist-test-task0waFvT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786512017613366-32144-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786512017613366-32144-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786512017613366-32144-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786512017613366-32144-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.100.62:39917
--webserver_interface=127.31.100.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786512017613366-32144-0/krb5kdc/oryx_127.31.100.62.keytab
--principal=oryx/127.31.100.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.100.20:32887
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.100.62:39917
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786512017613366-32144-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786512017613366-32144-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786512017613366-32144-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 05:20:52.441128 11982 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 05:20:52.441318 11982 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 05:20:52.441346 11982 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 05:20:52.442497 11982 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 05:20:52.442536 11982 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260812 05:20:52.442554 11982 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 05:20:52.442570 11982 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 05:20:52.442585 11982 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 05:20:52.442601 11982 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 05:20:52.444051 11982 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.100.20:32887
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786512017613366-32144-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786512017613366-32144-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.100.62:39917
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786512017613366-32144-0/krb5kdc/oryx_127.31.100.62.keytab
--principal=oryx/127.31.100.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.100.62:39917
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786512017613366-32144-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.100.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11982
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786512017613366-32144-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 04:20:34 UTC on ccycloud.quasar-phqdqp.root.comops.site
build id 4581871
I20260812 05:20:52.444289 11982 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 05:20:52.444480 11982 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 05:20:52.449029 11995 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 05:20:52.449043 11992 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 05:20:52.449096 11982 server_base.cc:1034] running on GCE node
W20260812 05:20:52.449044 11998 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 05:20:52.449517 11982 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 05:20:52.449756 11982 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 05:20:52.450884 11982 hybrid_clock.cc:648] HybridClock initialized: now 1786512052450872 us; error 30 us; skew 500 ppm
Aug 12 05:20:52 dist-test-slave-42z9 krb5kdc[11963](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512052, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.100.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 05:20:52.453187 11982 init.cc:377] Logged in from keytab as oryx/127.31.100.62@KRBTEST.COM (short username oryx)
I20260812 05:20:52.453636 11982 webserver.cc:460] Webserver started at http://127.31.100.62:38443/ using document root <none> and password file <none>
I20260812 05:20:52.453821 11982 fs_manager.cc:359] Metadata directory not provided
I20260812 05:20:52.453864 11982 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 05:20:52.453986 11982 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 05:20:52.454931 11982 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786512017613366-32144-0/minicluster-data/master-0/data/instance:
uuid: "f895c32b0dae4fdba2c12bfabf74b1cf"
format_stamp: "Formatted at 2026-08-12 05:20:52 on dist-test-slave-42z9"
server_key: "3d96e81c7959abbe51e27af5222ee441"
server_key_iv: "5983a28f90ec5e3ae6ad72e796bd2cbc"
server_key_version: "encryptionkey@0"
I20260812 05:20:52.455247 11982 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786512017613366-32144-0/minicluster-data/master-0/wal/instance:
uuid: "f895c32b0dae4fdba2c12bfabf74b1cf"
format_stamp: "Formatted at 2026-08-12 05:20:52 on dist-test-slave-42z9"
server_key: "3d96e81c7959abbe51e27af5222ee441"
server_key_iv: "5983a28f90ec5e3ae6ad72e796bd2cbc"
server_key_version: "encryptionkey@0"
I20260812 05:20:52.456494 11982 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260812 05:20:52.457119 12007 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 05:20:52.457255 11982 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260812 05:20:52.457336 11982 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786512017613366-32144-0/minicluster-data/master-0/data,/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786512017613366-32144-0/minicluster-data/master-0/wal
uuid: "f895c32b0dae4fdba2c12bfabf74b1cf"
format_stamp: "Formatted at 2026-08-12 05:20:52 on dist-test-slave-42z9"
server_key: "3d96e81c7959abbe51e27af5222ee441"
server_key_iv: "5983a28f90ec5e3ae6ad72e796bd2cbc"
server_key_version: "encryptionkey@0"
I20260812 05:20:52.457398 11982 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786512017613366-32144-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786512017613366-32144-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786512017613366-32144-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 05:20:52.474062 11982 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 05:20:52.476095 11982 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 05:20:52.476226 11982 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 05:20:52.480659 11982 rpc_server.cc:307] RPC server started. Bound to: 127.31.100.62:39917
I20260812 05:20:52.480707 12108 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.100.62:39917 every 8 connection(s)
I20260812 05:20:52.481004 11982 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786512017613366-32144-0/minicluster-data/master-0/data/info.pb
I20260812 05:20:52.481678 12109 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 05:20:52.483896 12109 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f895c32b0dae4fdba2c12bfabf74b1cf: Bootstrap starting.
I20260812 05:20:52.484467 12109 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f895c32b0dae4fdba2c12bfabf74b1cf: Neither blocks nor log segments found. Creating new log.
I20260812 05:20:52.484738 12109 log.cc:824] T 00000000000000000000000000000000 P f895c32b0dae4fdba2c12bfabf74b1cf: Log is configured to *not* fsync() on all Append() calls
I20260812 05:20:52.485414 12109 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f895c32b0dae4fdba2c12bfabf74b1cf: No bootstrap required, opened a new log
I20260812 05:20:52.486479 32144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0waFvT/build/release/bin/kudu as pid 11982
I20260812 05:20:52.486578 32144 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786512017613366-32144-0/minicluster-data/master-0/wal/instance
I20260812 05:20:52.486763 32144 external_mini_cluster.cc:1428] Setting key 17bcc236537381947bc850df0804ce6b
I20260812 05:20:52.487699 12109 raft_consensus.cc:348] T 00000000000000000000000000000000 P f895c32b0dae4fdba2c12bfabf74b1cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f895c32b0dae4fdba2c12bfabf74b1cf" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 39917 } }
I20260812 05:20:52.487829 12109 raft_consensus.cc:374] T 00000000000000000000000000000000 P f895c32b0dae4fdba2c12bfabf74b1cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 05:20:52.487861 12109 raft_consensus.cc:729] T 00000000000000000000000000000000 P f895c32b0dae4fdba2c12bfabf74b1cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f895c32b0dae4fdba2c12bfabf74b1cf, State: Initialized, Role: FOLLOWER
I20260812 05:20:52.487988 12109 consensus_queue.cc:260] T 00000000000000000000000000000000 P f895c32b0dae4fdba2c12bfabf74b1cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f895c32b0dae4fdba2c12bfabf74b1cf" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 39917 } }
I20260812 05:20:52.488050 12109 raft_consensus.cc:388] T 00000000000000000000000000000000 P f895c32b0dae4fdba2c12bfabf74b1cf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 05:20:52.488070 12109 raft_consensus.cc:482] T 00000000000000000000000000000000 P f895c32b0dae4fdba2c12bfabf74b1cf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 05:20:52.488101 12109 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f895c32b0dae4fdba2c12bfabf74b1cf [term 0 FOLLOWER]: Advancing to term 1
I20260812 05:20:52.488591 12109 raft_consensus.cc:504] T 00000000000000000000000000000000 P f895c32b0dae4fdba2c12bfabf74b1cf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f895c32b0dae4fdba2c12bfabf74b1cf" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 39917 } }
I20260812 05:20:52.488686 12109 leader_election.cc:302] T 00000000000000000000000000000000 P f895c32b0dae4fdba2c12bfabf74b1cf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f895c32b0dae4fdba2c12bfabf74b1cf; no voters:
I20260812 05:20:52.488839 12109 leader_election.cc:288] T 00000000000000000000000000000000 P f895c32b0dae4fdba2c12bfabf74b1cf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 05:20:52.488888 12119 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f895c32b0dae4fdba2c12bfabf74b1cf [term 1 FOLLOWER]: Leader election won for term 1
I20260812 05:20:52.489023 12119 raft_consensus.cc:686] T 00000000000000000000000000000000 P f895c32b0dae4fdba2c12bfabf74b1cf [term 1 LEADER]: Becoming Leader. State: Replica: f895c32b0dae4fdba2c12bfabf74b1cf, State: Running, Role: LEADER
I20260812 05:20:52.489094 12109 sys_catalog.cc:564] T 00000000000000000000000000000000 P f895c32b0dae4fdba2c12bfabf74b1cf [sys.catalog]: configured and running, proceeding with master startup.
I20260812 05:20:52.489128 12119 consensus_queue.cc:237] T 00000000000000000000000000000000 P f895c32b0dae4fdba2c12bfabf74b1cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f895c32b0dae4fdba2c12bfabf74b1cf" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 39917 } }
I20260812 05:20:52.489457 12119 sys_catalog.cc:455] T 00000000000000000000000000000000 P f895c32b0dae4fdba2c12bfabf74b1cf [sys.catalog]: SysCatalogTable state changed. Reason: New leader f895c32b0dae4fdba2c12bfabf74b1cf. Latest consensus state: current_term: 1 leader_uuid: "f895c32b0dae4fdba2c12bfabf74b1cf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f895c32b0dae4fdba2c12bfabf74b1cf" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 39917 } } }
I20260812 05:20:52.489542 12119 sys_catalog.cc:458] T 00000000000000000000000000000000 P f895c32b0dae4fdba2c12bfabf74b1cf [sys.catalog]: This master's current role is: LEADER
I20260812 05:20:52.489472 12120 sys_catalog.cc:455] T 00000000000000000000000000000000 P f895c32b0dae4fdba2c12bfabf74b1cf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f895c32b0dae4fdba2c12bfabf74b1cf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f895c32b0dae4fdba2c12bfabf74b1cf" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 39917 } } }
I20260812 05:20:52.489818 12120 sys_catalog.cc:458] T 00000000000000000000000000000000 P f895c32b0dae4fdba2c12bfabf74b1cf [sys.catalog]: This master's current role is: LEADER
I20260812 05:20:52.489718 12127 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 05:20:52.490422 12127 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 05:20:52.492305 12127 catalog_manager.cc:1347] Generated new cluster ID: 41e44a804ad843caa1c5ea7ef35826a0
I20260812 05:20:52.492358 12127 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 12 05:20:52 dist-test-slave-42z9 krb5kdc[11963](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512052, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.31.100.62@KRBTEST.COM
I20260812 05:20:52.497509 12127 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 05:20:52.497608 12118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:52.487245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57998 (local address 127.31.100.62:39917)
0812 05:20:52.487399 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0812 05:20:52.487409 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 05:20:52.487482 (+ 73us) server_negotiation.cc:407] Connection header received
0812 05:20:52.487660 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:20:52.487665 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:20:52.487707 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:20:52.487813 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:20:52.488471 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:52.489749 (+ 1278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:52.490769 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:52.491331 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:52.491440 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:52.494011 (+ 2571us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 05:20:52.494015 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 05:20:52.494015 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 05:20:52.494026 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0812 05:20:52.495792 (+ 1766us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:20:52.496149 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:20:52.496152 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:20:52.496153 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:20:52.496270 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:20:52.496465 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:20:52.496466 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:20:52.496466 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:20:52.496741 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 05:20:52.496869 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0812 05:20:52.497235 (+ 366us) server_negotiation.cc:299] Negotiation successful
0812 05:20:52.497371 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100,"thread_start_us":59,"threads_started":1}
I20260812 05:20:52.498070 12127 catalog_manager.cc:1504] Loading token signing keys...
I20260812 05:20:52.504493 12127 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f895c32b0dae4fdba2c12bfabf74b1cf: Generated new TSK 0
I20260812 05:20:52.504631 12127 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.31.100.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.31.100.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.31.100.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786512017613366-32144-0/krb5kdc/oryx_127.31.100.1.keytab.
Entry for principal oryx/127.31.100.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786512017613366-32144-0/krb5kdc/oryx_127.31.100.1.keytab.
I20260812 05:20:52.539410 32144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0waFvT/build/release/bin/kudu
/tmp/dist-test-task0waFvT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786512017613366-32144-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786512017613366-32144-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786512017613366-32144-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786512017613366-32144-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.100.1:0
--local_ip_for_outbound_sockets=127.31.100.1
--webserver_interface=127.31.100.1
--webserver_port=0
--tserver_master_addrs=127.31.100.62:39917
--keytab_file=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786512017613366-32144-0/krb5kdc/oryx_127.31.100.1.keytab
--principal=oryx/127.31.100.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.100.20:32887
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786512017613366-32144-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786512017613366-32144-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786512017613366-32144-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 05:20:52.643605 12152 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 05:20:52.643803 12152 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 05:20:52.643836 12152 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 05:20:52.644963 12152 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 05:20:52.645001 12152 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 05:20:52.645027 12152 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260812 05:20:52.645048 12152 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.100.1
I20260812 05:20:52.646636 12152 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.100.20:32887
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786512017613366-32144-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786512017613366-32144-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786512017613366-32144-0/krb5kdc/oryx_127.31.100.1.keytab
--principal=oryx/127.31.100.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.100.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786512017613366-32144-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.100.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.31.100.62:39917
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12152
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.100.1
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786512017613366-32144-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 04:20:34 UTC on ccycloud.quasar-phqdqp.root.comops.site
build id 4581871
I20260812 05:20:52.646862 12152 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 05:20:52.647081 12152 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 05:20:52.651716 12170 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 05:20:52.651690 12160 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 05:20:52.651703 12165 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 05:20:52.651873 12152 server_base.cc:1034] running on GCE node
I20260812 05:20:52.652114 12152 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 05:20:52.652339 12152 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 05:20:52.653522 12152 hybrid_clock.cc:648] HybridClock initialized: now 1786512052653503 us; error 35 us; skew 500 ppm
Aug 12 05:20:52 dist-test-slave-42z9 krb5kdc[11963](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512052, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.100.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 05:20:52.656013 12152 init.cc:377] Logged in from keytab as oryx/127.31.100.1@KRBTEST.COM (short username oryx)
I20260812 05:20:52.656464 12152 webserver.cc:460] Webserver started at http://127.31.100.1:39753/ using document root <none> and password file <none>
I20260812 05:20:52.656678 12152 fs_manager.cc:359] Metadata directory not provided
I20260812 05:20:52.656723 12152 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 05:20:52.656848 12152 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 05:20:52.657941 12152 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786512017613366-32144-0/minicluster-data/ts-0/data/instance:
uuid: "7c305125dc484bb0bcd6b6ccb48b8396"
format_stamp: "Formatted at 2026-08-12 05:20:52 on dist-test-slave-42z9"
server_key: "535fa7660f47487fb7ebcec9c48c7443"
server_key_iv: "109fe948cb1d0a1b01c881e2686c7539"
server_key_version: "encryptionkey@0"
I20260812 05:20:52.658273 12152 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786512017613366-32144-0/minicluster-data/ts-0/wal/instance:
uuid: "7c305125dc484bb0bcd6b6ccb48b8396"
format_stamp: "Formatted at 2026-08-12 05:20:52 on dist-test-slave-42z9"
server_key: "535fa7660f47487fb7ebcec9c48c7443"
server_key_iv: "109fe948cb1d0a1b01c881e2686c7539"
server_key_version: "encryptionkey@0"
I20260812 05:20:52.659663 12152 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260812 05:20:52.660475 12184 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 05:20:52.660647 12152 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 05:20:52.660751 12152 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786512017613366-32144-0/minicluster-data/ts-0/data,/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786512017613366-32144-0/minicluster-data/ts-0/wal
uuid: "7c305125dc484bb0bcd6b6ccb48b8396"
format_stamp: "Formatted at 2026-08-12 05:20:52 on dist-test-slave-42z9"
server_key: "535fa7660f47487fb7ebcec9c48c7443"
server_key_iv: "109fe948cb1d0a1b01c881e2686c7539"
server_key_version: "encryptionkey@0"
I20260812 05:20:52.660813 12152 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786512017613366-32144-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786512017613366-32144-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786512017613366-32144-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 05:20:52.679847 12152 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 05:20:52.681929 12152 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 05:20:52.682039 12152 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 05:20:52.682494 12152 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 05:20:52.682539 12152 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 05:20:52.682577 12152 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 05:20:52.682605 12152 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 12 05:20:52 dist-test-slave-42z9 krb5kdc[11963](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512052, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.100.1@KRBTEST.COM for oryx/127.31.100.62@KRBTEST.COM
I20260812 05:20:52.689260 12152 rpc_server.cc:307] RPC server started. Bound to: 127.31.100.1:46325
I20260812 05:20:52.689322 12393 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.100.1:46325 every 8 connection(s)
I20260812 05:20:52.689735 12152 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786512017613366-32144-0/minicluster-data/ts-0/data/info.pb
I20260812 05:20:52.692140 12118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:52.682934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.100.1:46963 (local address 127.31.100.62:39917)
0812 05:20:52.683021 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0812 05:20:52.683024 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 05:20:52.683272 (+ 248us) server_negotiation.cc:407] Connection header received
0812 05:20:52.683909 (+ 637us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:20:52.683910 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:20:52.683924 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:20:52.683979 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:20:52.684784 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:52.685582 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:52.687027 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:52.687339 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:52.687406 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:52.689763 (+ 2357us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 05:20:52.689764 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 05:20:52.689765 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 05:20:52.689771 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0812 05:20:52.691115 (+ 1344us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:20:52.691444 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:20:52.691444 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:20:52.691445 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:20:52.691490 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:20:52.691687 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:20:52.691688 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:20:52.691688 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:20:52.691785 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 05:20:52.691831 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0812 05:20:52.692018 (+ 187us) server_negotiation.cc:299] Negotiation successful
0812 05:20:52.692080 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260812 05:20:52.692296 12201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:52.682890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.62:39917 (local address 127.31.100.1:46963)
0812 05:20:52.683130 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0812 05:20:52.683179 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0812 05:20:52.683805 (+ 626us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:20:52.684051 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:20:52.684055 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:20:52.684415 (+ 360us) client_negotiation.cc:189] Negotiated authn=SASL
0812 05:20:52.684693 (+ 278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:20:52.684700 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:20:52.685676 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:20:52.685677 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:20:52.686943 (+ 1266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:20:52.686947 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:20:52.687415 (+ 468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:20:52.687416 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:20:52.687563 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:52.687980 (+ 417us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 05:20:52.687990 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0812 05:20:52.689684 (+ 1694us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 05:20:52.691181 (+ 1497us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 05:20:52.691187 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 05:20:52.691188 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 05:20:52.691390 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 05:20:52.691535 (+ 145us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 05:20:52.691536 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 05:20:52.691536 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 05:20:52.691642 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 05:20:52.691828 (+ 186us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 05:20:52.691831 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 05:20:52.691932 (+ 101us) client_negotiation.cc:769] Sending connection context
0812 05:20:52.692003 (+ 71us) client_negotiation.cc:240] Negotiation successful
0812 05:20:52.692118 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":72,"threads_started":1}
I20260812 05:20:52.692724 12394 heartbeater.cc:347] Connected to a master server at 127.31.100.62:39917
I20260812 05:20:52.692799 12394 heartbeater.cc:464] Registering TS with master...
I20260812 05:20:52.692960 12394 heartbeater.cc:511] Master 127.31.100.62:39917 requested a full tablet report, sending...
I20260812 05:20:52.693542 12031 ts_manager.cc:194] Registered new tserver with Master: 7c305125dc484bb0bcd6b6ccb48b8396 (127.31.100.1:46325)
I20260812 05:20:52.694185 12031 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.31.100.1@KRBTEST.COM'} at 127.31.100.1:46963
I20260812 05:20:52.694736 32144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0waFvT/build/release/bin/kudu as pid 12152
I20260812 05:20:52.694854 32144 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786512017613366-32144-0/minicluster-data/ts-0/wal/instance
I20260812 05:20:52.695075 32144 external_mini_cluster.cc:1428] Setting key 79758d4c256d62559dc1e4e3eea65e69
I20260812 05:20:52.699036 12118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:52.693705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.100.1:38737 (local address 127.31.100.62:39917)
0812 05:20:52.693747 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0812 05:20:52.693748 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0812 05:20:52.693864 (+ 116us) server_negotiation.cc:407] Connection header received
0812 05:20:52.693916 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:20:52.693917 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:20:52.693926 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:20:52.693970 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:20:52.694247 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:52.694888 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:52.695811 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:52.696082 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:52.696158 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:52.696900 (+ 742us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 05:20:52.696901 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 05:20:52.696901 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 05:20:52.696908 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 05:20:52.698096 (+ 1188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:20:52.698419 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:20:52.698420 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:20:52.698420 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:20:52.698460 (+ 40us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:20:52.698616 (+ 156us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:20:52.698617 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:20:52.698617 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:20:52.698696 (+ 79us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 05:20:52.698736 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0812 05:20:52.698883 (+ 147us) server_negotiation.cc:299] Negotiation successful
0812 05:20:52.698953 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260812 05:20:52.699049 12407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:52.693660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.62:39917 (local address 127.31.100.1:38737)
0812 05:20:52.693803 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0812 05:20:52.693812 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0812 05:20:52.693865 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:20:52.693974 (+ 109us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:20:52.693975 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:20:52.694141 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0812 05:20:52.694185 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:20:52.694187 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:20:52.695002 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:20:52.695003 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:20:52.695716 (+ 713us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:20:52.695719 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:20:52.696152 (+ 433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:20:52.696153 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:20:52.696203 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:52.696547 (+ 344us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 05:20:52.696556 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0812 05:20:52.696817 (+ 261us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 05:20:52.698166 (+ 1349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 05:20:52.698168 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 05:20:52.698168 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 05:20:52.698360 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 05:20:52.698507 (+ 147us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 05:20:52.698508 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 05:20:52.698508 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 05:20:52.698566 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 05:20:52.698742 (+ 176us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 05:20:52.698743 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 05:20:52.698788 (+ 45us) client_negotiation.cc:769] Sending connection context
0812 05:20:52.698852 (+ 64us) client_negotiation.cc:240] Negotiation successful
0812 05:20:52.698948 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":63,"threads_started":1}
WARNING: no policy specified for oryx/127.31.100.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.31.100.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.31.100.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786512017613366-32144-0/krb5kdc/oryx_127.31.100.2.keytab.
Entry for principal oryx/127.31.100.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786512017613366-32144-0/krb5kdc/oryx_127.31.100.2.keytab.
I20260812 05:20:52.721053 32144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0waFvT/build/release/bin/kudu
/tmp/dist-test-task0waFvT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786512017613366-32144-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786512017613366-32144-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786512017613366-32144-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786512017613366-32144-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.100.2:0
--local_ip_for_outbound_sockets=127.31.100.2
--webserver_interface=127.31.100.2
--webserver_port=0
--tserver_master_addrs=127.31.100.62:39917
--keytab_file=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786512017613366-32144-0/krb5kdc/oryx_127.31.100.2.keytab
--principal=oryx/127.31.100.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.100.20:32887
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786512017613366-32144-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786512017613366-32144-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786512017613366-32144-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 05:20:52.802335 12412 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 05:20:52.802536 12412 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 05:20:52.802558 12412 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 05:20:52.803635 12412 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 05:20:52.803668 12412 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 05:20:52.803686 12412 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260812 05:20:52.803702 12412 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.100.2
I20260812 05:20:52.805164 12412 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.100.20:32887
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786512017613366-32144-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786512017613366-32144-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786512017613366-32144-0/krb5kdc/oryx_127.31.100.2.keytab
--principal=oryx/127.31.100.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.100.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786512017613366-32144-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.100.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.31.100.62:39917
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12412
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.100.2
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786512017613366-32144-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 04:20:34 UTC on ccycloud.quasar-phqdqp.root.comops.site
build id 4581871
I20260812 05:20:52.805341 12412 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 05:20:52.805537 12412 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 05:20:52.810256 12417 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 05:20:52.810285 12420 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 05:20:52.810326 12412 server_base.cc:1034] running on GCE node
W20260812 05:20:52.810292 12418 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 05:20:52.810580 12412 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 05:20:52.810803 12412 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 05:20:52.811949 12412 hybrid_clock.cc:648] HybridClock initialized: now 1786512052811926 us; error 35 us; skew 500 ppm
Aug 12 05:20:52 dist-test-slave-42z9 krb5kdc[11963](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512052, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.100.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 05:20:52.814272 12412 init.cc:377] Logged in from keytab as oryx/127.31.100.2@KRBTEST.COM (short username oryx)
I20260812 05:20:52.814689 12412 webserver.cc:460] Webserver started at http://127.31.100.2:33619/ using document root <none> and password file <none>
I20260812 05:20:52.814884 12412 fs_manager.cc:359] Metadata directory not provided
I20260812 05:20:52.814934 12412 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 05:20:52.815054 12412 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 05:20:52.816035 12412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786512017613366-32144-0/minicluster-data/ts-1/data/instance:
uuid: "ff06107455ac4c638ba4dbd0061596d0"
format_stamp: "Formatted at 2026-08-12 05:20:52 on dist-test-slave-42z9"
server_key: "55b008dc33345fbc3dcf22660f9b7688"
server_key_iv: "1474689dee5b6ed6626ee2bd02ceba88"
server_key_version: "encryptionkey@0"
I20260812 05:20:52.816357 12412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786512017613366-32144-0/minicluster-data/ts-1/wal/instance:
uuid: "ff06107455ac4c638ba4dbd0061596d0"
format_stamp: "Formatted at 2026-08-12 05:20:52 on dist-test-slave-42z9"
server_key: "55b008dc33345fbc3dcf22660f9b7688"
server_key_iv: "1474689dee5b6ed6626ee2bd02ceba88"
server_key_version: "encryptionkey@0"
I20260812 05:20:52.817621 12412 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260812 05:20:52.818312 12431 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 05:20:52.818447 12412 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260812 05:20:52.818531 12412 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786512017613366-32144-0/minicluster-data/ts-1/data,/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786512017613366-32144-0/minicluster-data/ts-1/wal
uuid: "ff06107455ac4c638ba4dbd0061596d0"
format_stamp: "Formatted at 2026-08-12 05:20:52 on dist-test-slave-42z9"
server_key: "55b008dc33345fbc3dcf22660f9b7688"
server_key_iv: "1474689dee5b6ed6626ee2bd02ceba88"
server_key_version: "encryptionkey@0"
I20260812 05:20:52.818595 12412 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786512017613366-32144-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786512017613366-32144-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786512017613366-32144-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 05:20:52.842346 12412 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 05:20:52.843937 12412 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 05:20:52.844060 12412 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 05:20:52.844538 12412 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 05:20:52.844578 12412 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 05:20:52.844604 12412 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 05:20:52.844621 12412 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 12 05:20:52 dist-test-slave-42z9 krb5kdc[11963](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512052, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.100.2@KRBTEST.COM for oryx/127.31.100.62@KRBTEST.COM
I20260812 05:20:52.851202 12412 rpc_server.cc:307] RPC server started. Bound to: 127.31.100.2:37165
I20260812 05:20:52.851289 12627 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.100.2:37165 every 8 connection(s)
I20260812 05:20:52.851539 12412 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786512017613366-32144-0/minicluster-data/ts-1/data/info.pb
I20260812 05:20:52.853783 12118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:52.844905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.100.2:49725 (local address 127.31.100.62:39917)
0812 05:20:52.844975 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0812 05:20:52.844977 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 05:20:52.845130 (+ 153us) server_negotiation.cc:407] Connection header received
0812 05:20:52.845627 (+ 497us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:20:52.845628 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:20:52.845640 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:20:52.845712 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:20:52.846382 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:52.847113 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:52.848107 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:52.848303 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:52.848357 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:52.851226 (+ 2869us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 05:20:52.851228 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 05:20:52.851228 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 05:20:52.851236 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0812 05:20:52.852546 (+ 1310us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:20:52.852895 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:20:52.852897 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:20:52.852897 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:20:52.852953 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:20:52.853166 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:20:52.853167 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:20:52.853167 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:20:52.853275 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 05:20:52.853331 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0812 05:20:52.853618 (+ 287us) server_negotiation.cc:299] Negotiation successful
0812 05:20:52.853698 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260812 05:20:52.853940 12446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:52.844859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.62:39917 (local address 127.31.100.2:49725)
0812 05:20:52.845012 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0812 05:20:52.845050 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0812 05:20:52.845533 (+ 483us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:20:52.845785 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:20:52.845789 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:20:52.846119 (+ 330us) client_negotiation.cc:189] Negotiated authn=SASL
0812 05:20:52.846308 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:20:52.846312 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:20:52.847199 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:20:52.847201 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:20:52.848022 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:20:52.848025 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:20:52.848371 (+ 346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:20:52.848373 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:20:52.848523 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:52.849004 (+ 481us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 05:20:52.849013 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0812 05:20:52.851132 (+ 2119us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 05:20:52.852618 (+ 1486us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 05:20:52.852625 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 05:20:52.852627 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 05:20:52.852828 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 05:20:52.853008 (+ 180us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 05:20:52.853009 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 05:20:52.853009 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 05:20:52.853112 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 05:20:52.853330 (+ 218us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 05:20:52.853336 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 05:20:52.853492 (+ 156us) client_negotiation.cc:769] Sending connection context
0812 05:20:52.853587 (+ 95us) client_negotiation.cc:240] Negotiation successful
0812 05:20:52.853732 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":83,"threads_started":1}
I20260812 05:20:52.854368 12628 heartbeater.cc:347] Connected to a master server at 127.31.100.62:39917
I20260812 05:20:52.854470 12628 heartbeater.cc:464] Registering TS with master...
I20260812 05:20:52.854615 12628 heartbeater.cc:511] Master 127.31.100.62:39917 requested a full tablet report, sending...
I20260812 05:20:52.854979 12032 ts_manager.cc:194] Registered new tserver with Master: ff06107455ac4c638ba4dbd0061596d0 (127.31.100.2:37165)
I20260812 05:20:52.855438 12032 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.31.100.2@KRBTEST.COM'} at 127.31.100.2:49725
I20260812 05:20:52.855752 32144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0waFvT/build/release/bin/kudu as pid 12412
I20260812 05:20:52.855876 32144 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786512017613366-32144-0/minicluster-data/ts-1/wal/instance
I20260812 05:20:52.856098 32144 external_mini_cluster.cc:1428] Setting key 7f9a22f6191e759617e5084c25b15ca2
I20260812 05:20:52.860920 12635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:52.855240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.62:39917 (local address 127.31.100.2:54283)
0812 05:20:52.855347 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0812 05:20:52.855355 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0812 05:20:52.855397 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:20:52.855634 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:20:52.855635 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:20:52.855832 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0812 05:20:52.855907 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:20:52.855908 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:20:52.856699 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:20:52.856700 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:20:52.857356 (+ 656us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:20:52.857359 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:20:52.857720 (+ 361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:20:52.857722 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:20:52.857786 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:52.858121 (+ 335us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 05:20:52.858126 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0812 05:20:52.858455 (+ 329us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 05:20:52.859942 (+ 1487us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 05:20:52.859943 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 05:20:52.859943 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 05:20:52.860111 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 05:20:52.860316 (+ 205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 05:20:52.860317 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 05:20:52.860317 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 05:20:52.860372 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 05:20:52.860661 (+ 289us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 05:20:52.860662 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 05:20:52.860697 (+ 35us) client_negotiation.cc:769] Sending connection context
0812 05:20:52.860750 (+ 53us) client_negotiation.cc:240] Negotiation successful
0812 05:20:52.860857 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":83,"thread_start_us":61,"threads_started":1}
I20260812 05:20:52.860931 12118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:52.855307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.100.2:54283 (local address 127.31.100.62:39917)
0812 05:20:52.855537 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0812 05:20:52.855538 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0812 05:20:52.855545 (+ 7us) server_negotiation.cc:407] Connection header received
0812 05:20:52.855562 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:20:52.855563 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:20:52.855571 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:20:52.855628 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:20:52.856012 (+ 384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:52.856615 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:52.857462 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:52.857658 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:52.857709 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:52.858540 (+ 831us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 05:20:52.858542 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 05:20:52.858542 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 05:20:52.858549 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 05:20:52.859862 (+ 1313us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:20:52.860177 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:20:52.860178 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:20:52.860178 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:20:52.860238 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:20:52.860426 (+ 188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:20:52.860426 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:20:52.860427 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:20:52.860602 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 05:20:52.860669 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0812 05:20:52.860771 (+ 102us) server_negotiation.cc:299] Negotiation successful
0812 05:20:52.860840 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":216}
WARNING: no policy specified for oryx/127.31.100.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.31.100.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.31.100.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786512017613366-32144-0/krb5kdc/oryx_127.31.100.3.keytab.
Entry for principal oryx/127.31.100.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786512017613366-32144-0/krb5kdc/oryx_127.31.100.3.keytab.
I20260812 05:20:52.881305 32144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0waFvT/build/release/bin/kudu
/tmp/dist-test-task0waFvT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786512017613366-32144-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786512017613366-32144-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786512017613366-32144-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786512017613366-32144-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.100.3:0
--local_ip_for_outbound_sockets=127.31.100.3
--webserver_interface=127.31.100.3
--webserver_port=0
--tserver_master_addrs=127.31.100.62:39917
--keytab_file=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786512017613366-32144-0/krb5kdc/oryx_127.31.100.3.keytab
--principal=oryx/127.31.100.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.100.20:32887
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786512017613366-32144-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786512017613366-32144-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786512017613366-32144-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 05:20:52.960494 12638 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 05:20:52.960680 12638 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 05:20:52.960702 12638 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 05:20:52.961828 12638 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 05:20:52.961869 12638 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 05:20:52.961889 12638 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260812 05:20:52.961906 12638 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.100.3
I20260812 05:20:52.963346 12638 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.100.20:32887
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786512017613366-32144-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786512017613366-32144-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786512017613366-32144-0/krb5kdc/oryx_127.31.100.3.keytab
--principal=oryx/127.31.100.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.100.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786512017613366-32144-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.100.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.31.100.62:39917
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12638
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.100.3
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786512017613366-32144-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 04:20:34 UTC on ccycloud.quasar-phqdqp.root.comops.site
build id 4581871
I20260812 05:20:52.963555 12638 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 05:20:52.963727 12638 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 05:20:52.968279 12653 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 05:20:52.968295 12651 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 05:20:52.968276 12657 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 05:20:52.968464 12638 server_base.cc:1034] running on GCE node
I20260812 05:20:52.968647 12638 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 05:20:52.968822 12638 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 05:20:52.969955 12638 hybrid_clock.cc:648] HybridClock initialized: now 1786512052969943 us; error 26 us; skew 500 ppm
Aug 12 05:20:52 dist-test-slave-42z9 krb5kdc[11963](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512052, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.100.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 05:20:52.972225 12638 init.cc:377] Logged in from keytab as oryx/127.31.100.3@KRBTEST.COM (short username oryx)
I20260812 05:20:52.972632 12638 webserver.cc:460] Webserver started at http://127.31.100.3:33285/ using document root <none> and password file <none>
I20260812 05:20:52.972858 12638 fs_manager.cc:359] Metadata directory not provided
I20260812 05:20:52.972903 12638 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 05:20:52.973034 12638 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 05:20:52.974054 12638 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786512017613366-32144-0/minicluster-data/ts-2/data/instance:
uuid: "3280a541e7e9497686edd61460573e5f"
format_stamp: "Formatted at 2026-08-12 05:20:52 on dist-test-slave-42z9"
server_key: "8d934f807d71aacfaee598492e081591"
server_key_iv: "8912b5b98436c63c272719fadffcb8ca"
server_key_version: "encryptionkey@0"
I20260812 05:20:52.974359 12638 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786512017613366-32144-0/minicluster-data/ts-2/wal/instance:
uuid: "3280a541e7e9497686edd61460573e5f"
format_stamp: "Formatted at 2026-08-12 05:20:52 on dist-test-slave-42z9"
server_key: "8d934f807d71aacfaee598492e081591"
server_key_iv: "8912b5b98436c63c272719fadffcb8ca"
server_key_version: "encryptionkey@0"
I20260812 05:20:52.975541 12638 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260812 05:20:52.976166 12668 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 05:20:52.976313 12638 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260812 05:20:52.976394 12638 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786512017613366-32144-0/minicluster-data/ts-2/data,/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786512017613366-32144-0/minicluster-data/ts-2/wal
uuid: "3280a541e7e9497686edd61460573e5f"
format_stamp: "Formatted at 2026-08-12 05:20:52 on dist-test-slave-42z9"
server_key: "8d934f807d71aacfaee598492e081591"
server_key_iv: "8912b5b98436c63c272719fadffcb8ca"
server_key_version: "encryptionkey@0"
I20260812 05:20:52.976454 12638 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786512017613366-32144-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786512017613366-32144-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786512017613366-32144-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 05:20:52.999601 12638 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 05:20:53.001353 12638 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 05:20:53.001507 12638 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 05:20:53.002043 12638 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 05:20:53.002082 12638 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 05:20:53.002110 12638 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 05:20:53.002133 12638 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 12 05:20:53 dist-test-slave-42z9 krb5kdc[11963](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512052, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.100.3@KRBTEST.COM for oryx/127.31.100.62@KRBTEST.COM
I20260812 05:20:53.008574 12638 rpc_server.cc:307] RPC server started. Bound to: 127.31.100.3:35461
I20260812 05:20:53.008685 12856 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.100.3:35461 every 8 connection(s)
I20260812 05:20:53.008947 12638 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786512017613366-32144-0/minicluster-data/ts-2/data/info.pb
I20260812 05:20:53.011191 12118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:53.002338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.100.3:56157 (local address 127.31.100.62:39917)
0812 05:20:53.002386 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0812 05:20:53.002388 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 05:20:53.002618 (+ 230us) server_negotiation.cc:407] Connection header received
0812 05:20:53.003055 (+ 437us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:20:53.003057 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:20:53.003071 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:20:53.003148 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:20:53.003833 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:53.004617 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:53.005739 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:53.005952 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:53.006010 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:53.008866 (+ 2856us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 05:20:53.008868 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 05:20:53.008868 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 05:20:53.008875 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 05:20:53.010181 (+ 1306us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:20:53.010550 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:20:53.010551 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:20:53.010551 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:20:53.010596 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:20:53.010763 (+ 167us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:20:53.010764 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:20:53.010764 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:20:53.010852 (+ 88us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 05:20:53.010894 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0812 05:20:53.011070 (+ 176us) server_negotiation.cc:299] Negotiation successful
0812 05:20:53.011130 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260812 05:20:53.011351 12684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:53.002337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.62:39917 (local address 127.31.100.3:56157)
0812 05:20:53.002503 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0812 05:20:53.002543 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0812 05:20:53.002977 (+ 434us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:20:53.003200 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:20:53.003204 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:20:53.003475 (+ 271us) client_negotiation.cc:189] Negotiated authn=SASL
0812 05:20:53.003738 (+ 263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:20:53.003744 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:20:53.004708 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:20:53.004709 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:20:53.005665 (+ 956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:20:53.005669 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:20:53.006035 (+ 366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:20:53.006036 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:20:53.006173 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:53.006619 (+ 446us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 05:20:53.006628 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0812 05:20:53.008785 (+ 2157us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 05:20:53.010240 (+ 1455us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 05:20:53.010246 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 05:20:53.010248 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 05:20:53.010477 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 05:20:53.010633 (+ 156us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 05:20:53.010633 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 05:20:53.010634 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 05:20:53.010725 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 05:20:53.010886 (+ 161us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 05:20:53.010889 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 05:20:53.010987 (+ 98us) client_negotiation.cc:769] Sending connection context
0812 05:20:53.011055 (+ 68us) client_negotiation.cc:240] Negotiation successful
0812 05:20:53.011163 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":72,"threads_started":1}
I20260812 05:20:53.011794 12858 heartbeater.cc:347] Connected to a master server at 127.31.100.62:39917
I20260812 05:20:53.011874 12858 heartbeater.cc:464] Registering TS with master...
I20260812 05:20:53.012051 12858 heartbeater.cc:511] Master 127.31.100.62:39917 requested a full tablet report, sending...
I20260812 05:20:53.012418 12031 ts_manager.cc:194] Registered new tserver with Master: 3280a541e7e9497686edd61460573e5f (127.31.100.3:35461)
I20260812 05:20:53.012832 12031 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.31.100.3@KRBTEST.COM'} at 127.31.100.3:56157
I20260812 05:20:53.015687 32144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0waFvT/build/release/bin/kudu as pid 12638
I20260812 05:20:53.015782 32144 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786512017613366-32144-0/minicluster-data/ts-2/wal/instance
I20260812 05:20:53.015965 32144 external_mini_cluster.cc:1428] Setting key a7b965aa575b80e584cfb26304223fbb
I20260812 05:20:53.016661 32144 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 05:20:53.017576 12118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:53.012755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.100.3:49191 (local address 127.31.100.62:39917)
0812 05:20:53.012790 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0812 05:20:53.012792 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 05:20:53.012892 (+ 100us) server_negotiation.cc:407] Connection header received
0812 05:20:53.012950 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:20:53.012950 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:20:53.012959 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:20:53.013009 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:20:53.013334 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:53.013820 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:53.014435 (+ 615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:53.014617 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:53.014655 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:53.015335 (+ 680us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 05:20:53.015336 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 05:20:53.015336 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 05:20:53.015343 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 05:20:53.016596 (+ 1253us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:20:53.016913 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:20:53.016914 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:20:53.016914 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:20:53.016973 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:20:53.017129 (+ 156us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:20:53.017129 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:20:53.017130 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:20:53.017227 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 05:20:53.017279 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0812 05:20:53.017407 (+ 128us) server_negotiation.cc:299] Negotiation successful
0812 05:20:53.017492 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260812 05:20:53.017606 12869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:53.012704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.62:39917 (local address 127.31.100.3:49191)
0812 05:20:53.012833 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0812 05:20:53.012844 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0812 05:20:53.012906 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:20:53.013014 (+ 108us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:20:53.013015 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:20:53.013189 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0812 05:20:53.013264 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:20:53.013266 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:20:53.013894 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:20:53.013895 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:20:53.014374 (+ 479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:20:53.014376 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:20:53.014660 (+ 284us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:20:53.014661 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:20:53.014703 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:53.015026 (+ 323us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 05:20:53.015030 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0812 05:20:53.015267 (+ 237us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 05:20:53.016669 (+ 1402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 05:20:53.016670 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 05:20:53.016670 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 05:20:53.016846 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 05:20:53.017022 (+ 176us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 05:20:53.017023 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 05:20:53.017023 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 05:20:53.017078 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 05:20:53.017278 (+ 200us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 05:20:53.017279 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 05:20:53.017321 (+ 42us) client_negotiation.cc:769] Sending connection context
0812 05:20:53.017381 (+ 60us) client_negotiation.cc:240] Negotiation successful
0812 05:20:53.017521 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":73,"threads_started":1}
Aug 12 05:20:53 dist-test-slave-42z9 krb5kdc[11963](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512053, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 12 05:20:53 dist-test-slave-42z9 krb5kdc[11963](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.31.100.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 05:20:53 dist-test-slave-42z9 krb5kdc[11963](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.31.100.62@KRBTEST.COM, Server not found in Kerberos database
W20260812 05:20:53.037249 12118 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 05:20:53.032538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58014 (local address 127.31.100.62:39917)
0812 05:20:53.032590 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0812 05:20:53.032591 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0812 05:20:53.032599 (+ 8us) server_negotiation.cc:407] Connection header received
0812 05:20:53.032650 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:20:53.032651 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:20:53.032658 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:20:53.032709 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:20:53.033187 (+ 478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:53.033720 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:53.034408 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:53.034596 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:53.034645 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:53.037176 (+ 2531us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58014: BlockingRecv error: recv got EOF from 127.0.0.1:58014 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
Aug 12 05:20:53 dist-test-slave-42z9 krb5kdc[11963](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512053, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.31.100.62@KRBTEST.COM
I20260812 05:20:53.044062 12118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:53.037998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58030 (local address 127.31.100.62:39917)
0812 05:20:53.038062 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0812 05:20:53.038064 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 05:20:53.038075 (+ 11us) server_negotiation.cc:407] Connection header received
0812 05:20:53.038138 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:20:53.038139 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:20:53.038148 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:20:53.038192 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:20:53.038606 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:53.039052 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:53.039676 (+ 624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:53.039854 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:53.039903 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:53.041813 (+ 1910us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 05:20:53.041815 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 05:20:53.041815 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 05:20:53.041821 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0812 05:20:53.043036 (+ 1215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:20:53.043356 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:20:53.043356 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:20:53.043357 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:20:53.043406 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:20:53.043628 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:20:53.043628 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:20:53.043628 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:20:53.043723 (+ 95us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 05:20:53.043794 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0812 05:20:53.043944 (+ 150us) server_negotiation.cc:299] Negotiation successful
0812 05:20:53.044003 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260812 05:20:53.045399 12031 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58030:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 05:20:53.047025 12031 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 05:20:53.053253 12897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:53.049077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.1:46325 (local address 127.0.0.1:42818)
0812 05:20:53.049786 (+ 709us) negotiation.cc:107] Waiting for socket to connect
0812 05:20:53.049795 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0812 05:20:53.049834 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:20:53.050110 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:20:53.050111 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:20:53.050113 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 05:20:53.050154 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:20:53.050156 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:20:53.050984 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:20:53.050985 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:20:53.052274 (+ 1289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:20:53.052277 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:20:53.052962 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:20:53.052963 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:20:53.053052 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:53.053066 (+ 14us) client_negotiation.cc:769] Sending connection context
0812 05:20:53.053128 (+ 62us) client_negotiation.cc:240] Negotiation successful
0812 05:20:53.053182 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":694,"thread_start_us":96,"threads_started":1}
I20260812 05:20:53.053342 12899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:53.049070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.3:35461 (local address 127.0.0.1:41926)
0812 05:20:53.049410 (+ 340us) negotiation.cc:107] Waiting for socket to connect
0812 05:20:53.049420 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0812 05:20:53.049491 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:20:53.050118 (+ 627us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:20:53.050119 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:20:53.050122 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 05:20:53.050165 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:20:53.050167 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:20:53.051341 (+ 1174us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:20:53.051342 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:20:53.052245 (+ 903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:20:53.052247 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:20:53.053146 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:20:53.053147 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:20:53.053198 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:53.053201 (+ 3us) client_negotiation.cc:769] Sending connection context
0812 05:20:53.053254 (+ 53us) client_negotiation.cc:240] Negotiation successful
0812 05:20:53.053286 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":320,"mutex_wait_us":24,"thread_start_us":54,"threads_started":1}
I20260812 05:20:53.053407 12904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:53.049325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41926 (local address 127.31.100.3:35461)
0812 05:20:53.049958 (+ 633us) server_negotiation.cc:206] Beginning negotiation
0812 05:20:53.049963 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 05:20:53.049972 (+ 9us) server_negotiation.cc:407] Connection header received
0812 05:20:53.050006 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:20:53.050008 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:20:53.050022 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:20:53.050237 (+ 215us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 05:20:53.050294 (+ 57us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:53.051272 (+ 978us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:53.052317 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:53.053061 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:53.053139 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:53.053212 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0812 05:20:53.053267 (+ 55us) server_negotiation.cc:299] Negotiation successful
0812 05:20:53.053304 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":595,"thread_start_us":64,"threads_started":1}
I20260812 05:20:53.053602 12902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:53.049260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42818 (local address 127.31.100.1:46325)
0812 05:20:53.049477 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0812 05:20:53.049483 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 05:20:53.049929 (+ 446us) server_negotiation.cc:407] Connection header received
0812 05:20:53.049991 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:20:53.049993 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:20:53.050008 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:20:53.050220 (+ 212us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 05:20:53.050267 (+ 47us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:53.050909 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:53.052369 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:53.052876 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:53.052939 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:53.053019 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0812 05:20:53.053460 (+ 441us) server_negotiation.cc:299] Negotiation successful
0812 05:20:53.053506 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":168,"thread_start_us":91,"threads_started":1}
I20260812 05:20:53.053740 12895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:53.049069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.2:37165 (local address 127.0.0.1:35242)
0812 05:20:53.049223 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0812 05:20:53.049261 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0812 05:20:53.049323 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:20:53.049733 (+ 410us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:20:53.049735 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:20:53.049737 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 05:20:53.049793 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:20:53.049794 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:20:53.051178 (+ 1384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:20:53.051179 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:20:53.052277 (+ 1098us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:20:53.052278 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:20:53.053521 (+ 1243us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:20:53.053522 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:20:53.053582 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:53.053585 (+ 3us) client_negotiation.cc:769] Sending connection context
0812 05:20:53.053640 (+ 55us) client_negotiation.cc:240] Negotiation successful
0812 05:20:53.053679 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"mutex_wait_us":3,"thread_start_us":76,"threads_started":1}
I20260812 05:20:53.053809 12900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:53.049249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35242 (local address 127.31.100.2:37165)
0812 05:20:53.049598 (+ 349us) server_negotiation.cc:206] Beginning negotiation
0812 05:20:53.049610 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0812 05:20:53.049618 (+ 8us) server_negotiation.cc:407] Connection header received
0812 05:20:53.049648 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:20:53.049649 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:20:53.049661 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:20:53.049719 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 05:20:53.049873 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:53.051096 (+ 1223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:53.052390 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:53.053070 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:53.053139 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:53.053212 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0812 05:20:53.053669 (+ 457us) server_negotiation.cc:299] Negotiation successful
0812 05:20:53.053714 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":318,"thread_start_us":72,"threads_started":1}
I20260812 05:20:53.054165 12746 tablet_service.cc:1467] Processing CreateTablet for tablet 080999fb964c4e3aaa2b503dce1feb8a (DEFAULT_TABLE table=test-table [id=4286631aea9e4a4cb7ad5d8675bc78d6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 05:20:53.054239 12271 tablet_service.cc:1467] Processing CreateTablet for tablet 080999fb964c4e3aaa2b503dce1feb8a (DEFAULT_TABLE table=test-table [id=4286631aea9e4a4cb7ad5d8675bc78d6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 05:20:53.054427 12517 tablet_service.cc:1467] Processing CreateTablet for tablet 080999fb964c4e3aaa2b503dce1feb8a (DEFAULT_TABLE table=test-table [id=4286631aea9e4a4cb7ad5d8675bc78d6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 05:20:53.054556 12746 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 080999fb964c4e3aaa2b503dce1feb8a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 05:20:53.054600 12271 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 080999fb964c4e3aaa2b503dce1feb8a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 05:20:53.054736 12517 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 080999fb964c4e3aaa2b503dce1feb8a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 05:20:53.057873 12905 tablet_bootstrap.cc:492] T 080999fb964c4e3aaa2b503dce1feb8a P 3280a541e7e9497686edd61460573e5f: Bootstrap starting.
I20260812 05:20:53.058305 12907 tablet_bootstrap.cc:492] T 080999fb964c4e3aaa2b503dce1feb8a P ff06107455ac4c638ba4dbd0061596d0: Bootstrap starting.
I20260812 05:20:53.058475 12905 tablet_bootstrap.cc:654] T 080999fb964c4e3aaa2b503dce1feb8a P 3280a541e7e9497686edd61460573e5f: Neither blocks nor log segments found. Creating new log.
I20260812 05:20:53.058808 12905 log.cc:824] T 080999fb964c4e3aaa2b503dce1feb8a P 3280a541e7e9497686edd61460573e5f: Log is configured to *not* fsync() on all Append() calls
I20260812 05:20:53.058974 12907 tablet_bootstrap.cc:654] T 080999fb964c4e3aaa2b503dce1feb8a P ff06107455ac4c638ba4dbd0061596d0: Neither blocks nor log segments found. Creating new log.
I20260812 05:20:53.059263 12907 log.cc:824] T 080999fb964c4e3aaa2b503dce1feb8a P ff06107455ac4c638ba4dbd0061596d0: Log is configured to *not* fsync() on all Append() calls
I20260812 05:20:53.059661 12905 tablet_bootstrap.cc:492] T 080999fb964c4e3aaa2b503dce1feb8a P 3280a541e7e9497686edd61460573e5f: No bootstrap required, opened a new log
I20260812 05:20:53.059648 12906 tablet_bootstrap.cc:492] T 080999fb964c4e3aaa2b503dce1feb8a P 7c305125dc484bb0bcd6b6ccb48b8396: Bootstrap starting.
I20260812 05:20:53.059731 12905 ts_tablet_manager.cc:1397] T 080999fb964c4e3aaa2b503dce1feb8a P 3280a541e7e9497686edd61460573e5f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260812 05:20:53.059998 12907 tablet_bootstrap.cc:492] T 080999fb964c4e3aaa2b503dce1feb8a P ff06107455ac4c638ba4dbd0061596d0: No bootstrap required, opened a new log
I20260812 05:20:53.060058 12907 ts_tablet_manager.cc:1397] T 080999fb964c4e3aaa2b503dce1feb8a P ff06107455ac4c638ba4dbd0061596d0: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260812 05:20:53.060407 12906 tablet_bootstrap.cc:654] T 080999fb964c4e3aaa2b503dce1feb8a P 7c305125dc484bb0bcd6b6ccb48b8396: Neither blocks nor log segments found. Creating new log.
I20260812 05:20:53.060731 12906 log.cc:824] T 080999fb964c4e3aaa2b503dce1feb8a P 7c305125dc484bb0bcd6b6ccb48b8396: Log is configured to *not* fsync() on all Append() calls
I20260812 05:20:53.061551 12906 tablet_bootstrap.cc:492] T 080999fb964c4e3aaa2b503dce1feb8a P 7c305125dc484bb0bcd6b6ccb48b8396: No bootstrap required, opened a new log
I20260812 05:20:53.061622 12906 ts_tablet_manager.cc:1397] T 080999fb964c4e3aaa2b503dce1feb8a P 7c305125dc484bb0bcd6b6ccb48b8396: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260812 05:20:53.061826 12907 raft_consensus.cc:348] T 080999fb964c4e3aaa2b503dce1feb8a P ff06107455ac4c638ba4dbd0061596d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff06107455ac4c638ba4dbd0061596d0" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 37165 } } peers { permanent_uuid: "7c305125dc484bb0bcd6b6ccb48b8396" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 46325 } } peers { permanent_uuid: "3280a541e7e9497686edd61460573e5f" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 35461 } }
I20260812 05:20:53.061964 12907 raft_consensus.cc:374] T 080999fb964c4e3aaa2b503dce1feb8a P ff06107455ac4c638ba4dbd0061596d0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 05:20:53.061837 12905 raft_consensus.cc:348] T 080999fb964c4e3aaa2b503dce1feb8a P 3280a541e7e9497686edd61460573e5f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff06107455ac4c638ba4dbd0061596d0" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 37165 } } peers { permanent_uuid: "7c305125dc484bb0bcd6b6ccb48b8396" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 46325 } } peers { permanent_uuid: "3280a541e7e9497686edd61460573e5f" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 35461 } }
I20260812 05:20:53.062002 12907 raft_consensus.cc:729] T 080999fb964c4e3aaa2b503dce1feb8a P ff06107455ac4c638ba4dbd0061596d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff06107455ac4c638ba4dbd0061596d0, State: Initialized, Role: FOLLOWER
I20260812 05:20:53.062005 12905 raft_consensus.cc:374] T 080999fb964c4e3aaa2b503dce1feb8a P 3280a541e7e9497686edd61460573e5f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 05:20:53.062039 12905 raft_consensus.cc:729] T 080999fb964c4e3aaa2b503dce1feb8a P 3280a541e7e9497686edd61460573e5f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3280a541e7e9497686edd61460573e5f, State: Initialized, Role: FOLLOWER
I20260812 05:20:53.062131 12907 consensus_queue.cc:260] T 080999fb964c4e3aaa2b503dce1feb8a P ff06107455ac4c638ba4dbd0061596d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff06107455ac4c638ba4dbd0061596d0" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 37165 } } peers { permanent_uuid: "7c305125dc484bb0bcd6b6ccb48b8396" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 46325 } } peers { permanent_uuid: "3280a541e7e9497686edd61460573e5f" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 35461 } }
I20260812 05:20:53.062175 12905 consensus_queue.cc:260] T 080999fb964c4e3aaa2b503dce1feb8a P 3280a541e7e9497686edd61460573e5f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff06107455ac4c638ba4dbd0061596d0" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 37165 } } peers { permanent_uuid: "7c305125dc484bb0bcd6b6ccb48b8396" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 46325 } } peers { permanent_uuid: "3280a541e7e9497686edd61460573e5f" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 35461 } }
I20260812 05:20:53.062389 12907 ts_tablet_manager.cc:1428] T 080999fb964c4e3aaa2b503dce1feb8a P ff06107455ac4c638ba4dbd0061596d0: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260812 05:20:53.062515 12628 heartbeater.cc:503] Master 127.31.100.62:39917 was elected leader, sending a full tablet report...
I20260812 05:20:53.062521 12858 heartbeater.cc:503] Master 127.31.100.62:39917 was elected leader, sending a full tablet report...
I20260812 05:20:53.062445 12905 ts_tablet_manager.cc:1428] T 080999fb964c4e3aaa2b503dce1feb8a P 3280a541e7e9497686edd61460573e5f: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260812 05:20:53.063763 12906 raft_consensus.cc:348] T 080999fb964c4e3aaa2b503dce1feb8a P 7c305125dc484bb0bcd6b6ccb48b8396 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff06107455ac4c638ba4dbd0061596d0" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 37165 } } peers { permanent_uuid: "7c305125dc484bb0bcd6b6ccb48b8396" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 46325 } } peers { permanent_uuid: "3280a541e7e9497686edd61460573e5f" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 35461 } }
I20260812 05:20:53.063898 12906 raft_consensus.cc:374] T 080999fb964c4e3aaa2b503dce1feb8a P 7c305125dc484bb0bcd6b6ccb48b8396 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 05:20:53.063938 12906 raft_consensus.cc:729] T 080999fb964c4e3aaa2b503dce1feb8a P 7c305125dc484bb0bcd6b6ccb48b8396 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c305125dc484bb0bcd6b6ccb48b8396, State: Initialized, Role: FOLLOWER
I20260812 05:20:53.064026 12906 consensus_queue.cc:260] T 080999fb964c4e3aaa2b503dce1feb8a P 7c305125dc484bb0bcd6b6ccb48b8396 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff06107455ac4c638ba4dbd0061596d0" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 37165 } } peers { permanent_uuid: "7c305125dc484bb0bcd6b6ccb48b8396" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 46325 } } peers { permanent_uuid: "3280a541e7e9497686edd61460573e5f" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 35461 } }
I20260812 05:20:53.064257 12906 ts_tablet_manager.cc:1428] T 080999fb964c4e3aaa2b503dce1feb8a P 7c305125dc484bb0bcd6b6ccb48b8396: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260812 05:20:53.064303 12394 heartbeater.cc:503] Master 127.31.100.62:39917 was elected leader, sending a full tablet report...
W20260812 05:20:53.102092 12629 tablet.cc:2369] T 080999fb964c4e3aaa2b503dce1feb8a P ff06107455ac4c638ba4dbd0061596d0: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 05:20:53.190539 12399 tablet.cc:2369] T 080999fb964c4e3aaa2b503dce1feb8a P 7c305125dc484bb0bcd6b6ccb48b8396: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 05:20:53.190829 12917 raft_consensus.cc:482] T 080999fb964c4e3aaa2b503dce1feb8a P 3280a541e7e9497686edd61460573e5f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 05:20:53.190948 12917 raft_consensus.cc:504] T 080999fb964c4e3aaa2b503dce1feb8a P 3280a541e7e9497686edd61460573e5f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff06107455ac4c638ba4dbd0061596d0" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 37165 } } peers { permanent_uuid: "7c305125dc484bb0bcd6b6ccb48b8396" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 46325 } } peers { permanent_uuid: "3280a541e7e9497686edd61460573e5f" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 35461 } }
I20260812 05:20:53.191308 12917 leader_election.cc:288] T 080999fb964c4e3aaa2b503dce1feb8a P 3280a541e7e9497686edd61460573e5f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ff06107455ac4c638ba4dbd0061596d0 (127.31.100.2:37165), 7c305125dc484bb0bcd6b6ccb48b8396 (127.31.100.1:46325)
I20260812 05:20:53.194226 12684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:53.191512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.1:46325 (local address 127.31.100.3:33895)
0812 05:20:53.191573 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0812 05:20:53.191583 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0812 05:20:53.191632 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:20:53.191801 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:20:53.191802 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:20:53.191804 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 05:20:53.191878 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:20:53.191880 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:20:53.192623 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:20:53.192624 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:20:53.193475 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:20:53.193478 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:20:53.193990 (+ 512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:20:53.193991 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:20:53.194065 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:53.194068 (+ 3us) client_negotiation.cc:769] Sending connection context
0812 05:20:53.194112 (+ 44us) client_negotiation.cc:240] Negotiation successful
0812 05:20:53.194150 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260812 05:20:53.194226 12902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:53.191542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.100.3:33895 (local address 127.31.100.1:46325)
0812 05:20:53.191581 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0812 05:20:53.191583 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 05:20:53.191656 (+ 73us) server_negotiation.cc:407] Connection header received
0812 05:20:53.191703 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:20:53.191704 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:20:53.191716 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:20:53.191784 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 05:20:53.191972 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:53.192555 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:53.193554 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:53.193917 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:53.193988 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:53.193997 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0812 05:20:53.194128 (+ 131us) server_negotiation.cc:299] Negotiation successful
0812 05:20:53.194166 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260812 05:20:53.194386 12900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:53.191571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.100.3:35223 (local address 127.31.100.2:37165)
0812 05:20:53.191619 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0812 05:20:53.191621 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 05:20:53.191784 (+ 163us) server_negotiation.cc:407] Connection header received
0812 05:20:53.191859 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:20:53.191860 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:20:53.191868 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:20:53.191912 (+ 44us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 05:20:53.192121 (+ 209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:53.192635 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:53.193687 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:53.194055 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:53.194120 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:53.194128 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0812 05:20:53.194267 (+ 139us) server_negotiation.cc:299] Negotiation successful
0812 05:20:53.194306 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260812 05:20:53.194576 12923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:53.191513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.2:37165 (local address 127.31.100.3:35223)
0812 05:20:53.191726 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0812 05:20:53.191736 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0812 05:20:53.191810 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:20:53.191960 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:20:53.191961 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:20:53.191963 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 05:20:53.192060 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:20:53.192062 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:20:53.192709 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:20:53.192710 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:20:53.193608 (+ 898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:20:53.193611 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:20:53.194131 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:20:53.194132 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:20:53.194206 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:53.194210 (+ 4us) client_negotiation.cc:769] Sending connection context
0812 05:20:53.194255 (+ 45us) client_negotiation.cc:240] Negotiation successful
0812 05:20:53.194405 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":197,"mutex_wait_us":24,"thread_start_us":118,"threads_started":1}
I20260812 05:20:53.194604 12303 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "080999fb964c4e3aaa2b503dce1feb8a" candidate_uuid: "3280a541e7e9497686edd61460573e5f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c305125dc484bb0bcd6b6ccb48b8396" is_pre_election: true
I20260812 05:20:53.194741 12303 raft_consensus.cc:2454] T 080999fb964c4e3aaa2b503dce1feb8a P 7c305125dc484bb0bcd6b6ccb48b8396 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3280a541e7e9497686edd61460573e5f in term 0.
I20260812 05:20:53.194948 12546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "080999fb964c4e3aaa2b503dce1feb8a" candidate_uuid: "3280a541e7e9497686edd61460573e5f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff06107455ac4c638ba4dbd0061596d0" is_pre_election: true
I20260812 05:20:53.194999 12674 leader_election.cc:302] T 080999fb964c4e3aaa2b503dce1feb8a P 3280a541e7e9497686edd61460573e5f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3280a541e7e9497686edd61460573e5f, 7c305125dc484bb0bcd6b6ccb48b8396; no voters:
I20260812 05:20:53.195108 12546 raft_consensus.cc:2454] T 080999fb964c4e3aaa2b503dce1feb8a P ff06107455ac4c638ba4dbd0061596d0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3280a541e7e9497686edd61460573e5f in term 0.
I20260812 05:20:53.195175 12917 raft_consensus.cc:2781] T 080999fb964c4e3aaa2b503dce1feb8a P 3280a541e7e9497686edd61460573e5f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 05:20:53.195259 12917 raft_consensus.cc:482] T 080999fb964c4e3aaa2b503dce1feb8a P 3280a541e7e9497686edd61460573e5f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 05:20:53.195294 12917 raft_consensus.cc:3037] T 080999fb964c4e3aaa2b503dce1feb8a P 3280a541e7e9497686edd61460573e5f [term 0 FOLLOWER]: Advancing to term 1
I20260812 05:20:53.196280 12917 raft_consensus.cc:504] T 080999fb964c4e3aaa2b503dce1feb8a P 3280a541e7e9497686edd61460573e5f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff06107455ac4c638ba4dbd0061596d0" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 37165 } } peers { permanent_uuid: "7c305125dc484bb0bcd6b6ccb48b8396" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 46325 } } peers { permanent_uuid: "3280a541e7e9497686edd61460573e5f" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 35461 } }
I20260812 05:20:53.196410 12917 leader_election.cc:288] T 080999fb964c4e3aaa2b503dce1feb8a P 3280a541e7e9497686edd61460573e5f [CANDIDATE]: Term 1 election: Requested vote from peers ff06107455ac4c638ba4dbd0061596d0 (127.31.100.2:37165), 7c305125dc484bb0bcd6b6ccb48b8396 (127.31.100.1:46325)
I20260812 05:20:53.196554 12303 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "080999fb964c4e3aaa2b503dce1feb8a" candidate_uuid: "3280a541e7e9497686edd61460573e5f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c305125dc484bb0bcd6b6ccb48b8396"
I20260812 05:20:53.196623 12546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "080999fb964c4e3aaa2b503dce1feb8a" candidate_uuid: "3280a541e7e9497686edd61460573e5f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff06107455ac4c638ba4dbd0061596d0"
I20260812 05:20:53.196633 12303 raft_consensus.cc:3037] T 080999fb964c4e3aaa2b503dce1feb8a P 7c305125dc484bb0bcd6b6ccb48b8396 [term 0 FOLLOWER]: Advancing to term 1
I20260812 05:20:53.196705 12546 raft_consensus.cc:3037] T 080999fb964c4e3aaa2b503dce1feb8a P ff06107455ac4c638ba4dbd0061596d0 [term 0 FOLLOWER]: Advancing to term 1
I20260812 05:20:53.197291 12303 raft_consensus.cc:2454] T 080999fb964c4e3aaa2b503dce1feb8a P 7c305125dc484bb0bcd6b6ccb48b8396 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3280a541e7e9497686edd61460573e5f in term 1.
I20260812 05:20:53.197461 12546 raft_consensus.cc:2454] T 080999fb964c4e3aaa2b503dce1feb8a P ff06107455ac4c638ba4dbd0061596d0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3280a541e7e9497686edd61460573e5f in term 1.
I20260812 05:20:53.197489 12674 leader_election.cc:302] T 080999fb964c4e3aaa2b503dce1feb8a P 3280a541e7e9497686edd61460573e5f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3280a541e7e9497686edd61460573e5f, 7c305125dc484bb0bcd6b6ccb48b8396; no voters:
I20260812 05:20:53.197564 12917 raft_consensus.cc:2781] T 080999fb964c4e3aaa2b503dce1feb8a P 3280a541e7e9497686edd61460573e5f [term 1 FOLLOWER]: Leader election won for term 1
I20260812 05:20:53.197721 12917 raft_consensus.cc:686] T 080999fb964c4e3aaa2b503dce1feb8a P 3280a541e7e9497686edd61460573e5f [term 1 LEADER]: Becoming Leader. State: Replica: 3280a541e7e9497686edd61460573e5f, State: Running, Role: LEADER
I20260812 05:20:53.197808 12917 consensus_queue.cc:237] T 080999fb964c4e3aaa2b503dce1feb8a P 3280a541e7e9497686edd61460573e5f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff06107455ac4c638ba4dbd0061596d0" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 37165 } } peers { permanent_uuid: "7c305125dc484bb0bcd6b6ccb48b8396" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 46325 } } peers { permanent_uuid: "3280a541e7e9497686edd61460573e5f" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 35461 } }
I20260812 05:20:53.198627 12030 catalog_manager.cc:5495] T 080999fb964c4e3aaa2b503dce1feb8a P 3280a541e7e9497686edd61460573e5f reported cstate change: term changed from 0 to 1, leader changed from <none> to 3280a541e7e9497686edd61460573e5f (127.31.100.3). New cstate: current_term: 1 leader_uuid: "3280a541e7e9497686edd61460573e5f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff06107455ac4c638ba4dbd0061596d0" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 37165 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7c305125dc484bb0bcd6b6ccb48b8396" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 46325 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3280a541e7e9497686edd61460573e5f" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 35461 } health_report { overall_health: HEALTHY } } }
I20260812 05:20:53.232023 12904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:53.229675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41938 (local address 127.31.100.3:35461)
0812 05:20:53.230059 (+ 384us) server_negotiation.cc:206] Beginning negotiation
0812 05:20:53.230060 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0812 05:20:53.230068 (+ 8us) server_negotiation.cc:407] Connection header received
0812 05:20:53.230097 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:20:53.230098 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:20:53.230116 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:20:53.230172 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 05:20:53.230280 (+ 108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:53.230729 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:53.231436 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:53.231618 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:53.231670 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:53.231788 (+ 118us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 05:20:53.231882 (+ 94us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 05:20:53.231939 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0812 05:20:53.231956 (+ 17us) server_negotiation.cc:299] Negotiation successful
0812 05:20:53.231971 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":362}
I20260812 05:20:53.233686 12303 raft_consensus.cc:1261] T 080999fb964c4e3aaa2b503dce1feb8a P 7c305125dc484bb0bcd6b6ccb48b8396 [term 1 FOLLOWER]: Refusing update from remote peer 3280a541e7e9497686edd61460573e5f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 05:20:53.233686 12546 raft_consensus.cc:1261] T 080999fb964c4e3aaa2b503dce1feb8a P ff06107455ac4c638ba4dbd0061596d0 [term 1 FOLLOWER]: Refusing update from remote peer 3280a541e7e9497686edd61460573e5f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 05:20:53.233986 12917 consensus_queue.cc:1035] T 080999fb964c4e3aaa2b503dce1feb8a P 3280a541e7e9497686edd61460573e5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff06107455ac4c638ba4dbd0061596d0" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 37165 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 05:20:53.234133 12924 consensus_queue.cc:1035] T 080999fb964c4e3aaa2b503dce1feb8a P 3280a541e7e9497686edd61460573e5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "7c305125dc484bb0bcd6b6ccb48b8396" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 46325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 05:20:53.236212 12927 mvcc.cc:205] Tried to move back new op lower bound from 7317553370042699776 to 7317553369898864640. Current Snapshot: MvccSnapshot[applied={T|T < 7317553370042699776}]
I20260812 05:20:53.236479 12928 mvcc.cc:205] Tried to move back new op lower bound from 7317553370042699776 to 7317553369898864640. Current Snapshot: MvccSnapshot[applied={T|T < 7317553370042699776}]
I20260812 05:20:53.237128 12929 mvcc.cc:205] Tried to move back new op lower bound from 7317553370042699776 to 7317553369898864640. Current Snapshot: MvccSnapshot[applied={T|T < 7317553370042699776}]
I20260812 05:20:53.238483 12030 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58030:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 05:20:53.238583 12030 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58030:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 05:20:53.239286 12030 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f895c32b0dae4fdba2c12bfabf74b1cf: Sending DeleteTablet for 3 replicas of tablet 080999fb964c4e3aaa2b503dce1feb8a
I20260812 05:20:53.239624 12746 tablet_service.cc:1514] Processing DeleteTablet for tablet 080999fb964c4e3aaa2b503dce1feb8a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 05:20:53 UTC) from {username='oryx', principal='oryx/127.31.100.62@KRBTEST.COM'} at 127.0.0.1:41926
I20260812 05:20:53.239629 12517 tablet_service.cc:1514] Processing DeleteTablet for tablet 080999fb964c4e3aaa2b503dce1feb8a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 05:20:53 UTC) from {username='oryx', principal='oryx/127.31.100.62@KRBTEST.COM'} at 127.0.0.1:35242
I20260812 05:20:53.239737 12271 tablet_service.cc:1514] Processing DeleteTablet for tablet 080999fb964c4e3aaa2b503dce1feb8a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 05:20:53 UTC) from {username='oryx', principal='oryx/127.31.100.62@KRBTEST.COM'} at 127.0.0.1:42818
I20260812 05:20:53.239975 12943 tablet_replica.cc:330] T 080999fb964c4e3aaa2b503dce1feb8a P 7c305125dc484bb0bcd6b6ccb48b8396: stopping tablet replica
I20260812 05:20:53.240051 12942 tablet_replica.cc:330] T 080999fb964c4e3aaa2b503dce1feb8a P ff06107455ac4c638ba4dbd0061596d0: stopping tablet replica
I20260812 05:20:53.240140 12942 raft_consensus.cc:2229] T 080999fb964c4e3aaa2b503dce1feb8a P ff06107455ac4c638ba4dbd0061596d0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 05:20:53.240142 12030 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58030:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260812 05:20:53.240254 12942 raft_consensus.cc:2258] T 080999fb964c4e3aaa2b503dce1feb8a P ff06107455ac4c638ba4dbd0061596d0 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260812 05:20:53.240293 12030 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 05:20:53.240698 12942 ts_tablet_manager.cc:1905] T 080999fb964c4e3aaa2b503dce1feb8a P ff06107455ac4c638ba4dbd0061596d0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 05:20:53.240691 12943 raft_consensus.cc:2229] T 080999fb964c4e3aaa2b503dce1feb8a P 7c305125dc484bb0bcd6b6ccb48b8396 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 05:20:53.241196 12943 raft_consensus.cc:2258] T 080999fb964c4e3aaa2b503dce1feb8a P 7c305125dc484bb0bcd6b6ccb48b8396 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 05:20:53.241361 12940 tablet_replica.cc:330] T 080999fb964c4e3aaa2b503dce1feb8a P 3280a541e7e9497686edd61460573e5f: stopping tablet replica
I20260812 05:20:53.241472 12940 raft_consensus.cc:2229] T 080999fb964c4e3aaa2b503dce1feb8a P 3280a541e7e9497686edd61460573e5f [term 1 LEADER]: Raft consensus shutting down.
I20260812 05:20:53.241586 12940 raft_consensus.cc:2258] T 080999fb964c4e3aaa2b503dce1feb8a P 3280a541e7e9497686edd61460573e5f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 05:20:53.241622 12271 tablet_service.cc:1467] Processing CreateTablet for tablet ff9e2e977055461db36d37ff74022600 (DEFAULT_TABLE table=test-table [id=fcb113180ad44f32b31f57c34c265163]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 05:20:53.241731 12271 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ff9e2e977055461db36d37ff74022600. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 05:20:53.242018 12940 ts_tablet_manager.cc:1905] T 080999fb964c4e3aaa2b503dce1feb8a P 3280a541e7e9497686edd61460573e5f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 05:20:53.242661 12942 ts_tablet_manager.cc:1918] T 080999fb964c4e3aaa2b503dce1feb8a P ff06107455ac4c638ba4dbd0061596d0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 05:20:53.242725 12942 log.cc:1196] T 080999fb964c4e3aaa2b503dce1feb8a P ff06107455ac4c638ba4dbd0061596d0: Deleting WAL directory at /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786512017613366-32144-0/minicluster-data/ts-1/wal/wals/080999fb964c4e3aaa2b503dce1feb8a
I20260812 05:20:53.242988 12942 ts_tablet_manager.cc:1939] T 080999fb964c4e3aaa2b503dce1feb8a P ff06107455ac4c638ba4dbd0061596d0: Deleting consensus metadata
I20260812 05:20:53.243101 12906 tablet_bootstrap.cc:492] T ff9e2e977055461db36d37ff74022600 P 7c305125dc484bb0bcd6b6ccb48b8396: Bootstrap starting.
I20260812 05:20:53.243353 12943 ts_tablet_manager.cc:1905] T 080999fb964c4e3aaa2b503dce1feb8a P 7c305125dc484bb0bcd6b6ccb48b8396: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 05:20:53.243350 12008 catalog_manager.cc:4841] TS ff06107455ac4c638ba4dbd0061596d0 (127.31.100.2:37165): tablet 080999fb964c4e3aaa2b503dce1feb8a (table test-table [id=4286631aea9e4a4cb7ad5d8675bc78d6]) successfully deleted
I20260812 05:20:53.243492 12906 tablet_bootstrap.cc:654] T ff9e2e977055461db36d37ff74022600 P 7c305125dc484bb0bcd6b6ccb48b8396: Neither blocks nor log segments found. Creating new log.
I20260812 05:20:53.243496 12746 tablet_service.cc:1467] Processing CreateTablet for tablet ff9e2e977055461db36d37ff74022600 (DEFAULT_TABLE table=test-table [id=fcb113180ad44f32b31f57c34c265163]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 05:20:53.243498 12517 tablet_service.cc:1467] Processing CreateTablet for tablet ff9e2e977055461db36d37ff74022600 (DEFAULT_TABLE table=test-table [id=fcb113180ad44f32b31f57c34c265163]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 05:20:53.243592 12517 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ff9e2e977055461db36d37ff74022600. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 05:20:53.243592 12746 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ff9e2e977055461db36d37ff74022600. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 05:20:53.244256 12940 ts_tablet_manager.cc:1918] T 080999fb964c4e3aaa2b503dce1feb8a P 3280a541e7e9497686edd61460573e5f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 05:20:53.244318 12940 log.cc:1196] T 080999fb964c4e3aaa2b503dce1feb8a P 3280a541e7e9497686edd61460573e5f: Deleting WAL directory at /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786512017613366-32144-0/minicluster-data/ts-2/wal/wals/080999fb964c4e3aaa2b503dce1feb8a
I20260812 05:20:53.244544 12940 ts_tablet_manager.cc:1939] T 080999fb964c4e3aaa2b503dce1feb8a P 3280a541e7e9497686edd61460573e5f: Deleting consensus metadata
I20260812 05:20:53.244678 12907 tablet_bootstrap.cc:492] T ff9e2e977055461db36d37ff74022600 P ff06107455ac4c638ba4dbd0061596d0: Bootstrap starting.
I20260812 05:20:53.244717 12943 ts_tablet_manager.cc:1918] T 080999fb964c4e3aaa2b503dce1feb8a P 7c305125dc484bb0bcd6b6ccb48b8396: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 05:20:53.244774 12943 log.cc:1196] T 080999fb964c4e3aaa2b503dce1feb8a P 7c305125dc484bb0bcd6b6ccb48b8396: Deleting WAL directory at /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786512017613366-32144-0/minicluster-data/ts-0/wal/wals/080999fb964c4e3aaa2b503dce1feb8a
I20260812 05:20:53.244853 12905 tablet_bootstrap.cc:492] T ff9e2e977055461db36d37ff74022600 P 3280a541e7e9497686edd61460573e5f: Bootstrap starting.
I20260812 05:20:53.245002 12943 ts_tablet_manager.cc:1939] T 080999fb964c4e3aaa2b503dce1feb8a P 7c305125dc484bb0bcd6b6ccb48b8396: Deleting consensus metadata
I20260812 05:20:53.245059 12907 tablet_bootstrap.cc:654] T ff9e2e977055461db36d37ff74022600 P ff06107455ac4c638ba4dbd0061596d0: Neither blocks nor log segments found. Creating new log.
I20260812 05:20:53.245260 12905 tablet_bootstrap.cc:654] T ff9e2e977055461db36d37ff74022600 P 3280a541e7e9497686edd61460573e5f: Neither blocks nor log segments found. Creating new log.
I20260812 05:20:53.245764 12906 tablet_bootstrap.cc:492] T ff9e2e977055461db36d37ff74022600 P 7c305125dc484bb0bcd6b6ccb48b8396: No bootstrap required, opened a new log
I20260812 05:20:53.245767 12013 catalog_manager.cc:4841] TS 7c305125dc484bb0bcd6b6ccb48b8396 (127.31.100.1:46325): tablet 080999fb964c4e3aaa2b503dce1feb8a (table test-table [id=4286631aea9e4a4cb7ad5d8675bc78d6]) successfully deleted
I20260812 05:20:53.245819 12906 ts_tablet_manager.cc:1397] T ff9e2e977055461db36d37ff74022600 P 7c305125dc484bb0bcd6b6ccb48b8396: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260812 05:20:53.246052 12907 tablet_bootstrap.cc:492] T ff9e2e977055461db36d37ff74022600 P ff06107455ac4c638ba4dbd0061596d0: No bootstrap required, opened a new log
I20260812 05:20:53.246109 12907 ts_tablet_manager.cc:1397] T ff9e2e977055461db36d37ff74022600 P ff06107455ac4c638ba4dbd0061596d0: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260812 05:20:53.246176 12906 raft_consensus.cc:348] T ff9e2e977055461db36d37ff74022600 P 7c305125dc484bb0bcd6b6ccb48b8396 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c305125dc484bb0bcd6b6ccb48b8396" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 46325 } } peers { permanent_uuid: "ff06107455ac4c638ba4dbd0061596d0" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 37165 } } peers { permanent_uuid: "3280a541e7e9497686edd61460573e5f" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 35461 } }
I20260812 05:20:53.246248 12906 raft_consensus.cc:374] T ff9e2e977055461db36d37ff74022600 P 7c305125dc484bb0bcd6b6ccb48b8396 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 05:20:53.246268 12906 raft_consensus.cc:729] T ff9e2e977055461db36d37ff74022600 P 7c305125dc484bb0bcd6b6ccb48b8396 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c305125dc484bb0bcd6b6ccb48b8396, State: Initialized, Role: FOLLOWER
I20260812 05:20:53.246250 12907 raft_consensus.cc:348] T ff9e2e977055461db36d37ff74022600 P ff06107455ac4c638ba4dbd0061596d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c305125dc484bb0bcd6b6ccb48b8396" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 46325 } } peers { permanent_uuid: "ff06107455ac4c638ba4dbd0061596d0" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 37165 } } peers { permanent_uuid: "3280a541e7e9497686edd61460573e5f" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 35461 } }
I20260812 05:20:53.246304 12907 raft_consensus.cc:374] T ff9e2e977055461db36d37ff74022600 P ff06107455ac4c638ba4dbd0061596d0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 05:20:53.246322 12907 raft_consensus.cc:729] T ff9e2e977055461db36d37ff74022600 P ff06107455ac4c638ba4dbd0061596d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff06107455ac4c638ba4dbd0061596d0, State: Initialized, Role: FOLLOWER
I20260812 05:20:53.246304 12906 consensus_queue.cc:260] T ff9e2e977055461db36d37ff74022600 P 7c305125dc484bb0bcd6b6ccb48b8396 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c305125dc484bb0bcd6b6ccb48b8396" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 46325 } } peers { permanent_uuid: "ff06107455ac4c638ba4dbd0061596d0" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 37165 } } peers { permanent_uuid: "3280a541e7e9497686edd61460573e5f" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 35461 } }
I20260812 05:20:53.246361 12907 consensus_queue.cc:260] T ff9e2e977055461db36d37ff74022600 P ff06107455ac4c638ba4dbd0061596d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c305125dc484bb0bcd6b6ccb48b8396" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 46325 } } peers { permanent_uuid: "ff06107455ac4c638ba4dbd0061596d0" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 37165 } } peers { permanent_uuid: "3280a541e7e9497686edd61460573e5f" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 35461 } }
I20260812 05:20:53.246411 12906 ts_tablet_manager.cc:1428] T ff9e2e977055461db36d37ff74022600 P 7c305125dc484bb0bcd6b6ccb48b8396: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260812 05:20:53.246448 12907 ts_tablet_manager.cc:1428] T ff9e2e977055461db36d37ff74022600 P ff06107455ac4c638ba4dbd0061596d0: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260812 05:20:53.246760 12011 catalog_manager.cc:4841] TS 3280a541e7e9497686edd61460573e5f (127.31.100.3:35461): tablet 080999fb964c4e3aaa2b503dce1feb8a (table test-table [id=4286631aea9e4a4cb7ad5d8675bc78d6]) successfully deleted
I20260812 05:20:53.246953 12905 tablet_bootstrap.cc:492] T ff9e2e977055461db36d37ff74022600 P 3280a541e7e9497686edd61460573e5f: No bootstrap required, opened a new log
I20260812 05:20:53.246986 12905 ts_tablet_manager.cc:1397] T ff9e2e977055461db36d37ff74022600 P 3280a541e7e9497686edd61460573e5f: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260812 05:20:53.247114 12905 raft_consensus.cc:348] T ff9e2e977055461db36d37ff74022600 P 3280a541e7e9497686edd61460573e5f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c305125dc484bb0bcd6b6ccb48b8396" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 46325 } } peers { permanent_uuid: "ff06107455ac4c638ba4dbd0061596d0" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 37165 } } peers { permanent_uuid: "3280a541e7e9497686edd61460573e5f" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 35461 } }
I20260812 05:20:53.247197 12905 raft_consensus.cc:374] T ff9e2e977055461db36d37ff74022600 P 3280a541e7e9497686edd61460573e5f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 05:20:53.247216 12905 raft_consensus.cc:729] T ff9e2e977055461db36d37ff74022600 P 3280a541e7e9497686edd61460573e5f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3280a541e7e9497686edd61460573e5f, State: Initialized, Role: FOLLOWER
I20260812 05:20:53.247246 12905 consensus_queue.cc:260] T ff9e2e977055461db36d37ff74022600 P 3280a541e7e9497686edd61460573e5f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c305125dc484bb0bcd6b6ccb48b8396" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 46325 } } peers { permanent_uuid: "ff06107455ac4c638ba4dbd0061596d0" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 37165 } } peers { permanent_uuid: "3280a541e7e9497686edd61460573e5f" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 35461 } }
I20260812 05:20:53.247323 12905 ts_tablet_manager.cc:1428] T ff9e2e977055461db36d37ff74022600 P 3280a541e7e9497686edd61460573e5f: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
W20260812 05:20:53.259478 12859 tablet.cc:2369] T ff9e2e977055461db36d37ff74022600 P 3280a541e7e9497686edd61460573e5f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 05:20:53.287396 12915 raft_consensus.cc:482] T ff9e2e977055461db36d37ff74022600 P ff06107455ac4c638ba4dbd0061596d0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 05:20:53.287503 12915 raft_consensus.cc:504] T ff9e2e977055461db36d37ff74022600 P ff06107455ac4c638ba4dbd0061596d0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c305125dc484bb0bcd6b6ccb48b8396" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 46325 } } peers { permanent_uuid: "ff06107455ac4c638ba4dbd0061596d0" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 37165 } } peers { permanent_uuid: "3280a541e7e9497686edd61460573e5f" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 35461 } }
I20260812 05:20:53.287807 12915 leader_election.cc:288] T ff9e2e977055461db36d37ff74022600 P ff06107455ac4c638ba4dbd0061596d0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7c305125dc484bb0bcd6b6ccb48b8396 (127.31.100.1:46325), 3280a541e7e9497686edd61460573e5f (127.31.100.3:35461)
I20260812 05:20:53.291581 12902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:53.288087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.100.2:55545 (local address 127.31.100.1:46325)
0812 05:20:53.288129 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0812 05:20:53.288131 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 05:20:53.288231 (+ 100us) server_negotiation.cc:407] Connection header received
0812 05:20:53.288316 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:20:53.288318 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:20:53.288327 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:20:53.288374 (+ 47us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 05:20:53.288546 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:53.289086 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:53.290625 (+ 1539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:53.291226 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:53.291299 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:53.291308 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0812 05:20:53.291477 (+ 169us) server_negotiation.cc:299] Negotiation successful
0812 05:20:53.291518 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260812 05:20:53.291602 12950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:53.287983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.1:46325 (local address 127.31.100.2:55545)
0812 05:20:53.288175 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0812 05:20:53.288185 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0812 05:20:53.288255 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:20:53.288417 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:20:53.288418 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:20:53.288419 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 05:20:53.288492 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:20:53.288494 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:20:53.289151 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:20:53.289152 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:20:53.290545 (+ 1393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:20:53.290549 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:20:53.291315 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:20:53.291316 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:20:53.291393 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:53.291398 (+ 5us) client_negotiation.cc:769] Sending connection context
0812 05:20:53.291455 (+ 57us) client_negotiation.cc:240] Negotiation successful
0812 05:20:53.291498 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":180,"mutex_wait_us":68,"thread_start_us":61,"threads_started":1}
I20260812 05:20:53.291800 12303 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff9e2e977055461db36d37ff74022600" candidate_uuid: "ff06107455ac4c638ba4dbd0061596d0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c305125dc484bb0bcd6b6ccb48b8396" is_pre_election: true
I20260812 05:20:53.291879 12303 raft_consensus.cc:2454] T ff9e2e977055461db36d37ff74022600 P 7c305125dc484bb0bcd6b6ccb48b8396 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff06107455ac4c638ba4dbd0061596d0 in term 0.
I20260812 05:20:53.292053 12438 leader_election.cc:302] T ff9e2e977055461db36d37ff74022600 P ff06107455ac4c638ba4dbd0061596d0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7c305125dc484bb0bcd6b6ccb48b8396, ff06107455ac4c638ba4dbd0061596d0; no voters:
I20260812 05:20:53.292177 12915 raft_consensus.cc:2781] T ff9e2e977055461db36d37ff74022600 P ff06107455ac4c638ba4dbd0061596d0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 05:20:53.292207 12915 raft_consensus.cc:482] T ff9e2e977055461db36d37ff74022600 P ff06107455ac4c638ba4dbd0061596d0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 05:20:53.292223 12915 raft_consensus.cc:3037] T ff9e2e977055461db36d37ff74022600 P ff06107455ac4c638ba4dbd0061596d0 [term 0 FOLLOWER]: Advancing to term 1
I20260812 05:20:53.292349 12904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:53.288082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.100.2:35951 (local address 127.31.100.3:35461)
0812 05:20:53.288129 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0812 05:20:53.288131 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 05:20:53.288200 (+ 69us) server_negotiation.cc:407] Connection header received
0812 05:20:53.288274 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:20:53.288274 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:20:53.288300 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:20:53.288349 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 05:20:53.288508 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:53.289260 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:53.291342 (+ 2082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:53.291872 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:53.291952 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:53.291959 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0812 05:20:53.292255 (+ 296us) server_negotiation.cc:299] Negotiation successful
0812 05:20:53.292293 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260812 05:20:53.292502 12446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:53.287982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.3:35461 (local address 127.31.100.2:35951)
0812 05:20:53.288146 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0812 05:20:53.288157 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0812 05:20:53.288203 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:20:53.288366 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:20:53.288367 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:20:53.288369 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 05:20:53.288436 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:20:53.288438 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:20:53.289326 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:20:53.289327 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:20:53.290557 (+ 1230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:20:53.290559 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:20:53.292024 (+ 1465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:20:53.292025 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:20:53.292107 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:53.292110 (+ 3us) client_negotiation.cc:769] Sending connection context
0812 05:20:53.292161 (+ 51us) client_negotiation.cc:240] Negotiation successful
0812 05:20:53.292212 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"mutex_wait_us":31}
I20260812 05:20:53.292722 12781 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff9e2e977055461db36d37ff74022600" candidate_uuid: "ff06107455ac4c638ba4dbd0061596d0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3280a541e7e9497686edd61460573e5f" is_pre_election: true
I20260812 05:20:53.292853 12781 raft_consensus.cc:2454] T ff9e2e977055461db36d37ff74022600 P 3280a541e7e9497686edd61460573e5f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff06107455ac4c638ba4dbd0061596d0 in term 0.
I20260812 05:20:53.293082 12915 raft_consensus.cc:504] T ff9e2e977055461db36d37ff74022600 P ff06107455ac4c638ba4dbd0061596d0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c305125dc484bb0bcd6b6ccb48b8396" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 46325 } } peers { permanent_uuid: "ff06107455ac4c638ba4dbd0061596d0" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 37165 } } peers { permanent_uuid: "3280a541e7e9497686edd61460573e5f" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 35461 } }
I20260812 05:20:53.293210 12915 leader_election.cc:288] T ff9e2e977055461db36d37ff74022600 P ff06107455ac4c638ba4dbd0061596d0 [CANDIDATE]: Term 1 election: Requested vote from peers 7c305125dc484bb0bcd6b6ccb48b8396 (127.31.100.1:46325), 3280a541e7e9497686edd61460573e5f (127.31.100.3:35461)
I20260812 05:20:53.293416 12781 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff9e2e977055461db36d37ff74022600" candidate_uuid: "ff06107455ac4c638ba4dbd0061596d0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3280a541e7e9497686edd61460573e5f"
I20260812 05:20:53.293416 12303 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff9e2e977055461db36d37ff74022600" candidate_uuid: "ff06107455ac4c638ba4dbd0061596d0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c305125dc484bb0bcd6b6ccb48b8396"
I20260812 05:20:53.293506 12781 raft_consensus.cc:3037] T ff9e2e977055461db36d37ff74022600 P 3280a541e7e9497686edd61460573e5f [term 0 FOLLOWER]: Advancing to term 1
I20260812 05:20:53.293514 12303 raft_consensus.cc:3037] T ff9e2e977055461db36d37ff74022600 P 7c305125dc484bb0bcd6b6ccb48b8396 [term 0 FOLLOWER]: Advancing to term 1
I20260812 05:20:53.294092 12303 raft_consensus.cc:2454] T ff9e2e977055461db36d37ff74022600 P 7c305125dc484bb0bcd6b6ccb48b8396 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff06107455ac4c638ba4dbd0061596d0 in term 1.
I20260812 05:20:53.294281 12781 raft_consensus.cc:2454] T ff9e2e977055461db36d37ff74022600 P 3280a541e7e9497686edd61460573e5f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff06107455ac4c638ba4dbd0061596d0 in term 1.
I20260812 05:20:53.294456 12436 leader_election.cc:302] T ff9e2e977055461db36d37ff74022600 P ff06107455ac4c638ba4dbd0061596d0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3280a541e7e9497686edd61460573e5f, ff06107455ac4c638ba4dbd0061596d0; no voters:
I20260812 05:20:53.294577 12915 raft_consensus.cc:2781] T ff9e2e977055461db36d37ff74022600 P ff06107455ac4c638ba4dbd0061596d0 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 05:20:53.294731 12915 raft_consensus.cc:686] T ff9e2e977055461db36d37ff74022600 P ff06107455ac4c638ba4dbd0061596d0 [term 1 LEADER]: Becoming Leader. State: Replica: ff06107455ac4c638ba4dbd0061596d0, State: Running, Role: LEADER
I20260812 05:20:53.294809 12915 consensus_queue.cc:237] T ff9e2e977055461db36d37ff74022600 P ff06107455ac4c638ba4dbd0061596d0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c305125dc484bb0bcd6b6ccb48b8396" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 46325 } } peers { permanent_uuid: "ff06107455ac4c638ba4dbd0061596d0" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 37165 } } peers { permanent_uuid: "3280a541e7e9497686edd61460573e5f" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 35461 } }
I20260812 05:20:53.295444 12030 catalog_manager.cc:5495] T ff9e2e977055461db36d37ff74022600 P ff06107455ac4c638ba4dbd0061596d0 reported cstate change: term changed from 0 to 1, leader changed from <none> to ff06107455ac4c638ba4dbd0061596d0 (127.31.100.2). New cstate: current_term: 1 leader_uuid: "ff06107455ac4c638ba4dbd0061596d0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c305125dc484bb0bcd6b6ccb48b8396" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 46325 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff06107455ac4c638ba4dbd0061596d0" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 37165 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3280a541e7e9497686edd61460573e5f" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 35461 } health_report { overall_health: UNKNOWN } } }
Aug 12 05:20:53 dist-test-slave-42z9 krb5kdc[11963](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512052, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.100.62@KRBTEST.COM for oryx/127.31.100.62@KRBTEST.COM
I20260812 05:20:53.325711 12118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:53.318130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58034 (local address 127.31.100.62:39917)
0812 05:20:53.318175 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0812 05:20:53.318177 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 05:20:53.318289 (+ 112us) server_negotiation.cc:407] Connection header received
0812 05:20:53.318353 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:20:53.318354 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:20:53.318366 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:20:53.318444 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:20:53.319016 (+ 572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:53.319525 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:53.320496 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:53.320679 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:53.320737 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:53.323280 (+ 2543us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 05:20:53.323281 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 05:20:53.323281 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 05:20:53.323287 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0812 05:20:53.324513 (+ 1226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:20:53.324936 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:20:53.324937 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:20:53.324937 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:20:53.324982 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:20:53.325165 (+ 183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:20:53.325166 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:20:53.325166 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:20:53.325251 (+ 85us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 05:20:53.325292 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0812 05:20:53.325498 (+ 206us) server_negotiation.cc:299] Negotiation successful
0812 05:20:53.325622 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":55,"server-negotiator.queue_time_us":25}
I20260812 05:20:53.325709 12962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:53.318048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.62:39917 (local address 127.0.0.1:58034)
0812 05:20:53.318210 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0812 05:20:53.318222 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0812 05:20:53.318296 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:20:53.318459 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:20:53.318460 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:20:53.318864 (+ 404us) client_negotiation.cc:189] Negotiated authn=SASL
0812 05:20:53.318931 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:20:53.318934 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:20:53.319604 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:20:53.319605 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:20:53.320401 (+ 796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:20:53.320404 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:20:53.320756 (+ 352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:20:53.320757 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:20:53.320819 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:53.321496 (+ 677us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 05:20:53.321501 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0812 05:20:53.323196 (+ 1695us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 05:20:53.324591 (+ 1395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 05:20:53.324599 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 05:20:53.324601 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 05:20:53.324858 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 05:20:53.325036 (+ 178us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 05:20:53.325036 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 05:20:53.325037 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 05:20:53.325111 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 05:20:53.325304 (+ 193us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 05:20:53.325310 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 05:20:53.325361 (+ 51us) client_negotiation.cc:769] Sending connection context
0812 05:20:53.325458 (+ 97us) client_negotiation.cc:240] Negotiation successful
0812 05:20:53.325570 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":83,"threads_started":1}
I20260812 05:20:53.327755 12030 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.31.100.62@KRBTEST.COM'} at 127.0.0.1:58034:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260812 05:20:53.327952 12030 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 05:20:53.329275 12746 tablet_service.cc:1467] Processing CreateTablet for tablet 1bcaf6e3854a4fff9e8bfbcf1fd476eb (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=47e63e37660642509a27639b5b77da19]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260812 05:20:53.329267 12271 tablet_service.cc:1467] Processing CreateTablet for tablet 1bcaf6e3854a4fff9e8bfbcf1fd476eb (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=47e63e37660642509a27639b5b77da19]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260812 05:20:53.329286 12517 tablet_service.cc:1467] Processing CreateTablet for tablet 1bcaf6e3854a4fff9e8bfbcf1fd476eb (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=47e63e37660642509a27639b5b77da19]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260812 05:20:53.329417 12746 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1bcaf6e3854a4fff9e8bfbcf1fd476eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 05:20:53.329418 12271 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1bcaf6e3854a4fff9e8bfbcf1fd476eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 05:20:53.329432 12517 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1bcaf6e3854a4fff9e8bfbcf1fd476eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 05:20:53.330866 12905 tablet_bootstrap.cc:492] T 1bcaf6e3854a4fff9e8bfbcf1fd476eb P 3280a541e7e9497686edd61460573e5f: Bootstrap starting.
I20260812 05:20:53.330919 12907 tablet_bootstrap.cc:492] T 1bcaf6e3854a4fff9e8bfbcf1fd476eb P ff06107455ac4c638ba4dbd0061596d0: Bootstrap starting.
I20260812 05:20:53.331229 12905 tablet_bootstrap.cc:654] T 1bcaf6e3854a4fff9e8bfbcf1fd476eb P 3280a541e7e9497686edd61460573e5f: Neither blocks nor log segments found. Creating new log.
I20260812 05:20:53.331303 12907 tablet_bootstrap.cc:654] T 1bcaf6e3854a4fff9e8bfbcf1fd476eb P ff06107455ac4c638ba4dbd0061596d0: Neither blocks nor log segments found. Creating new log.
I20260812 05:20:53.331671 12906 tablet_bootstrap.cc:492] T 1bcaf6e3854a4fff9e8bfbcf1fd476eb P 7c305125dc484bb0bcd6b6ccb48b8396: Bootstrap starting.
I20260812 05:20:53.331981 12907 tablet_bootstrap.cc:492] T 1bcaf6e3854a4fff9e8bfbcf1fd476eb P ff06107455ac4c638ba4dbd0061596d0: No bootstrap required, opened a new log
I20260812 05:20:53.332023 12906 tablet_bootstrap.cc:654] T 1bcaf6e3854a4fff9e8bfbcf1fd476eb P 7c305125dc484bb0bcd6b6ccb48b8396: Neither blocks nor log segments found. Creating new log.
I20260812 05:20:53.332024 12907 ts_tablet_manager.cc:1397] T 1bcaf6e3854a4fff9e8bfbcf1fd476eb P ff06107455ac4c638ba4dbd0061596d0: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260812 05:20:53.332285 12907 raft_consensus.cc:348] T 1bcaf6e3854a4fff9e8bfbcf1fd476eb P ff06107455ac4c638ba4dbd0061596d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c305125dc484bb0bcd6b6ccb48b8396" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 46325 } } peers { permanent_uuid: "ff06107455ac4c638ba4dbd0061596d0" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 37165 } } peers { permanent_uuid: "3280a541e7e9497686edd61460573e5f" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 35461 } }
I20260812 05:20:53.332350 12907 raft_consensus.cc:374] T 1bcaf6e3854a4fff9e8bfbcf1fd476eb P ff06107455ac4c638ba4dbd0061596d0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 05:20:53.332367 12907 raft_consensus.cc:729] T 1bcaf6e3854a4fff9e8bfbcf1fd476eb P ff06107455ac4c638ba4dbd0061596d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff06107455ac4c638ba4dbd0061596d0, State: Initialized, Role: FOLLOWER
I20260812 05:20:53.332418 12907 consensus_queue.cc:260] T 1bcaf6e3854a4fff9e8bfbcf1fd476eb P ff06107455ac4c638ba4dbd0061596d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c305125dc484bb0bcd6b6ccb48b8396" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 46325 } } peers { permanent_uuid: "ff06107455ac4c638ba4dbd0061596d0" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 37165 } } peers { permanent_uuid: "3280a541e7e9497686edd61460573e5f" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 35461 } }
I20260812 05:20:53.332513 12907 ts_tablet_manager.cc:1428] T 1bcaf6e3854a4fff9e8bfbcf1fd476eb P ff06107455ac4c638ba4dbd0061596d0: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260812 05:20:53.332556 12906 tablet_bootstrap.cc:492] T 1bcaf6e3854a4fff9e8bfbcf1fd476eb P 7c305125dc484bb0bcd6b6ccb48b8396: No bootstrap required, opened a new log
I20260812 05:20:53.332609 12906 ts_tablet_manager.cc:1397] T 1bcaf6e3854a4fff9e8bfbcf1fd476eb P 7c305125dc484bb0bcd6b6ccb48b8396: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260812 05:20:53.332644 12915 tablet_replica.cc:439] T 1bcaf6e3854a4fff9e8bfbcf1fd476eb P ff06107455ac4c638ba4dbd0061596d0: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c305125dc484bb0bcd6b6ccb48b8396" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 46325 } } peers { permanent_uuid: "ff06107455ac4c638ba4dbd0061596d0" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 37165 } } peers { permanent_uuid: "3280a541e7e9497686edd61460573e5f" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 35461 } } }
I20260812 05:20:53.332713 12915 tablet_replica.cc:442] T 1bcaf6e3854a4fff9e8bfbcf1fd476eb P ff06107455ac4c638ba4dbd0061596d0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 05:20:53.332742 12906 raft_consensus.cc:348] T 1bcaf6e3854a4fff9e8bfbcf1fd476eb P 7c305125dc484bb0bcd6b6ccb48b8396 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c305125dc484bb0bcd6b6ccb48b8396" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 46325 } } peers { permanent_uuid: "ff06107455ac4c638ba4dbd0061596d0" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 37165 } } peers { permanent_uuid: "3280a541e7e9497686edd61460573e5f" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 35461 } }
I20260812 05:20:53.332813 12906 raft_consensus.cc:374] T 1bcaf6e3854a4fff9e8bfbcf1fd476eb P 7c305125dc484bb0bcd6b6ccb48b8396 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 05:20:53.332834 12906 raft_consensus.cc:729] T 1bcaf6e3854a4fff9e8bfbcf1fd476eb P 7c305125dc484bb0bcd6b6ccb48b8396 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c305125dc484bb0bcd6b6ccb48b8396, State: Initialized, Role: FOLLOWER
I20260812 05:20:53.332872 12906 consensus_queue.cc:260] T 1bcaf6e3854a4fff9e8bfbcf1fd476eb P 7c305125dc484bb0bcd6b6ccb48b8396 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c305125dc484bb0bcd6b6ccb48b8396" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 46325 } } peers { permanent_uuid: "ff06107455ac4c638ba4dbd0061596d0" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 37165 } } peers { permanent_uuid: "3280a541e7e9497686edd61460573e5f" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 35461 } }
I20260812 05:20:53.332907 12905 tablet_bootstrap.cc:492] T 1bcaf6e3854a4fff9e8bfbcf1fd476eb P 3280a541e7e9497686edd61460573e5f: No bootstrap required, opened a new log
I20260812 05:20:53.332962 12905 ts_tablet_manager.cc:1397] T 1bcaf6e3854a4fff9e8bfbcf1fd476eb P 3280a541e7e9497686edd61460573e5f: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260812 05:20:53.333038 12906 ts_tablet_manager.cc:1428] T 1bcaf6e3854a4fff9e8bfbcf1fd476eb P 7c305125dc484bb0bcd6b6ccb48b8396: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260812 05:20:53.333106 12905 raft_consensus.cc:348] T 1bcaf6e3854a4fff9e8bfbcf1fd476eb P 3280a541e7e9497686edd61460573e5f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c305125dc484bb0bcd6b6ccb48b8396" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 46325 } } peers { permanent_uuid: "ff06107455ac4c638ba4dbd0061596d0" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 37165 } } peers { permanent_uuid: "3280a541e7e9497686edd61460573e5f" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 35461 } }
I20260812 05:20:53.333159 12905 raft_consensus.cc:374] T 1bcaf6e3854a4fff9e8bfbcf1fd476eb P 3280a541e7e9497686edd61460573e5f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 05:20:53.333173 12905 raft_consensus.cc:729] T 1bcaf6e3854a4fff9e8bfbcf1fd476eb P 3280a541e7e9497686edd61460573e5f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3280a541e7e9497686edd61460573e5f, State: Initialized, Role: FOLLOWER
I20260812 05:20:53.333209 12905 consensus_queue.cc:260] T 1bcaf6e3854a4fff9e8bfbcf1fd476eb P 3280a541e7e9497686edd61460573e5f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c305125dc484bb0bcd6b6ccb48b8396" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 46325 } } peers { permanent_uuid: "ff06107455ac4c638ba4dbd0061596d0" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 37165 } } peers { permanent_uuid: "3280a541e7e9497686edd61460573e5f" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 35461 } }
I20260812 05:20:53.333290 12905 ts_tablet_manager.cc:1428] T 1bcaf6e3854a4fff9e8bfbcf1fd476eb P 3280a541e7e9497686edd61460573e5f: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260812 05:20:53.333324 12932 tablet_replica.cc:439] T 1bcaf6e3854a4fff9e8bfbcf1fd476eb P 3280a541e7e9497686edd61460573e5f: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c305125dc484bb0bcd6b6ccb48b8396" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 46325 } } peers { permanent_uuid: "ff06107455ac4c638ba4dbd0061596d0" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 37165 } } peers { permanent_uuid: "3280a541e7e9497686edd61460573e5f" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 35461 } } }
I20260812 05:20:53.333374 12920 raft_consensus.cc:482] T 1bcaf6e3854a4fff9e8bfbcf1fd476eb P 7c305125dc484bb0bcd6b6ccb48b8396 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 05:20:53.333389 12932 tablet_replica.cc:442] T 1bcaf6e3854a4fff9e8bfbcf1fd476eb P 3280a541e7e9497686edd61460573e5f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 05:20:53.333415 12920 raft_consensus.cc:504] T 1bcaf6e3854a4fff9e8bfbcf1fd476eb P 7c305125dc484bb0bcd6b6ccb48b8396 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c305125dc484bb0bcd6b6ccb48b8396" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 46325 } } peers { permanent_uuid: "ff06107455ac4c638ba4dbd0061596d0" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 37165 } } peers { permanent_uuid: "3280a541e7e9497686edd61460573e5f" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 35461 } }
I20260812 05:20:53.333768 12920 leader_election.cc:288] T 1bcaf6e3854a4fff9e8bfbcf1fd476eb P 7c305125dc484bb0bcd6b6ccb48b8396 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ff06107455ac4c638ba4dbd0061596d0 (127.31.100.2:37165), 3280a541e7e9497686edd61460573e5f (127.31.100.3:35461)
I20260812 05:20:53.333884 12920 tablet_replica.cc:439] T 1bcaf6e3854a4fff9e8bfbcf1fd476eb P 7c305125dc484bb0bcd6b6ccb48b8396: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c305125dc484bb0bcd6b6ccb48b8396" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 46325 } } peers { permanent_uuid: "ff06107455ac4c638ba4dbd0061596d0" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 37165 } } peers { permanent_uuid: "3280a541e7e9497686edd61460573e5f" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 35461 } } }
I20260812 05:20:53.333958 12920 tablet_replica.cc:442] T 1bcaf6e3854a4fff9e8bfbcf1fd476eb P 7c305125dc484bb0bcd6b6ccb48b8396: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 05:20:53.337105 12900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:53.334080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.100.1:33989 (local address 127.31.100.2:37165)
0812 05:20:53.334137 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0812 05:20:53.334139 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 05:20:53.334332 (+ 193us) server_negotiation.cc:407] Connection header received
0812 05:20:53.334407 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:20:53.334408 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:20:53.334417 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:20:53.334462 (+ 45us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 05:20:53.334932 (+ 470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:53.335377 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:53.336354 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:53.336800 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:53.336836 (+ 36us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:53.336842 (+ 6us) server_negotiation.cc:1035] Waiting for connection context
0812 05:20:53.337009 (+ 167us) server_negotiation.cc:299] Negotiation successful
0812 05:20:53.337046 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260812 05:20:53.337293 12971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:53.333889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.2:37165 (local address 127.31.100.1:33989)
0812 05:20:53.334272 (+ 383us) negotiation.cc:107] Waiting for socket to connect
0812 05:20:53.334283 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0812 05:20:53.334335 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:20:53.334769 (+ 434us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:20:53.334771 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:20:53.334774 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 05:20:53.334849 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:20:53.334851 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:20:53.335479 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:20:53.335480 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:20:53.336236 (+ 756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:20:53.336238 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:20:53.336888 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:20:53.336889 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:20:53.336943 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:53.336947 (+ 4us) client_negotiation.cc:769] Sending connection context
0812 05:20:53.336987 (+ 40us) client_negotiation.cc:240] Negotiation successful
0812 05:20:53.337025 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":357,"thread_start_us":105,"threads_started":1}
I20260812 05:20:53.337540 12546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1bcaf6e3854a4fff9e8bfbcf1fd476eb" candidate_uuid: "7c305125dc484bb0bcd6b6ccb48b8396" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff06107455ac4c638ba4dbd0061596d0" is_pre_election: true
I20260812 05:20:53.337620 12546 raft_consensus.cc:2454] T 1bcaf6e3854a4fff9e8bfbcf1fd476eb P ff06107455ac4c638ba4dbd0061596d0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7c305125dc484bb0bcd6b6ccb48b8396 in term 0.
I20260812 05:20:53.337775 12186 leader_election.cc:302] T 1bcaf6e3854a4fff9e8bfbcf1fd476eb P 7c305125dc484bb0bcd6b6ccb48b8396 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7c305125dc484bb0bcd6b6ccb48b8396, ff06107455ac4c638ba4dbd0061596d0; no voters:
I20260812 05:20:53.337954 12969 raft_consensus.cc:2781] T 1bcaf6e3854a4fff9e8bfbcf1fd476eb P 7c305125dc484bb0bcd6b6ccb48b8396 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 05:20:53.338033 12969 raft_consensus.cc:482] T 1bcaf6e3854a4fff9e8bfbcf1fd476eb P 7c305125dc484bb0bcd6b6ccb48b8396 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 05:20:53.338088 12904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:53.334081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.100.1:46979 (local address 127.31.100.3:35461)
0812 05:20:53.334170 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0812 05:20:53.334172 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 05:20:53.335025 (+ 853us) server_negotiation.cc:407] Connection header received
0812 05:20:53.335115 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:20:53.335116 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:20:53.335127 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:20:53.335156 (+ 29us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 05:20:53.335331 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:53.335990 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:53.337268 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:53.337795 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:53.337859 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:53.337866 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0812 05:20:53.337994 (+ 128us) server_negotiation.cc:299] Negotiation successful
0812 05:20:53.338028 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260812 05:20:53.338182 12974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:53.333898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.3:35461 (local address 127.31.100.1:46979)
0812 05:20:53.334970 (+ 1072us) negotiation.cc:107] Waiting for socket to connect
0812 05:20:53.334980 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0812 05:20:53.335050 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:20:53.335211 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:20:53.335211 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:20:53.335213 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 05:20:53.335265 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:20:53.335267 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:20:53.336087 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:20:53.336088 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:20:53.337182 (+ 1094us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:20:53.337185 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:20:53.337867 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:20:53.337868 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:20:53.337938 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:53.337942 (+ 4us) client_negotiation.cc:769] Sending connection context
0812 05:20:53.337989 (+ 47us) client_negotiation.cc:240] Negotiation successful
0812 05:20:53.338027 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1052,"thread_start_us":97,"threads_started":1}
I20260812 05:20:53.338340 12969 raft_consensus.cc:3037] T 1bcaf6e3854a4fff9e8bfbcf1fd476eb P 7c305125dc484bb0bcd6b6ccb48b8396 [term 0 FOLLOWER]: Advancing to term 1
I20260812 05:20:53.338557 12781 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1bcaf6e3854a4fff9e8bfbcf1fd476eb" candidate_uuid: "7c305125dc484bb0bcd6b6ccb48b8396" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3280a541e7e9497686edd61460573e5f" is_pre_election: true
I20260812 05:20:53.338625 12781 raft_consensus.cc:2454] T 1bcaf6e3854a4fff9e8bfbcf1fd476eb P 3280a541e7e9497686edd61460573e5f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7c305125dc484bb0bcd6b6ccb48b8396 in term 0.
I20260812 05:20:53.339226 12969 raft_consensus.cc:504] T 1bcaf6e3854a4fff9e8bfbcf1fd476eb P 7c305125dc484bb0bcd6b6ccb48b8396 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c305125dc484bb0bcd6b6ccb48b8396" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 46325 } } peers { permanent_uuid: "ff06107455ac4c638ba4dbd0061596d0" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 37165 } } peers { permanent_uuid: "3280a541e7e9497686edd61460573e5f" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 35461 } }
I20260812 05:20:53.339489 12969 leader_election.cc:288] T 1bcaf6e3854a4fff9e8bfbcf1fd476eb P 7c305125dc484bb0bcd6b6ccb48b8396 [CANDIDATE]: Term 1 election: Requested vote from peers ff06107455ac4c638ba4dbd0061596d0 (127.31.100.2:37165), 3280a541e7e9497686edd61460573e5f (127.31.100.3:35461)
I20260812 05:20:53.339481 12546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1bcaf6e3854a4fff9e8bfbcf1fd476eb" candidate_uuid: "7c305125dc484bb0bcd6b6ccb48b8396" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff06107455ac4c638ba4dbd0061596d0"
I20260812 05:20:53.339521 12781 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1bcaf6e3854a4fff9e8bfbcf1fd476eb" candidate_uuid: "7c305125dc484bb0bcd6b6ccb48b8396" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3280a541e7e9497686edd61460573e5f"
I20260812 05:20:53.339550 12546 raft_consensus.cc:3037] T 1bcaf6e3854a4fff9e8bfbcf1fd476eb P ff06107455ac4c638ba4dbd0061596d0 [term 0 FOLLOWER]: Advancing to term 1
I20260812 05:20:53.339572 12781 raft_consensus.cc:3037] T 1bcaf6e3854a4fff9e8bfbcf1fd476eb P 3280a541e7e9497686edd61460573e5f [term 0 FOLLOWER]: Advancing to term 1
I20260812 05:20:53.340190 12781 raft_consensus.cc:2454] T 1bcaf6e3854a4fff9e8bfbcf1fd476eb P 3280a541e7e9497686edd61460573e5f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7c305125dc484bb0bcd6b6ccb48b8396 in term 1.
I20260812 05:20:53.340190 12546 raft_consensus.cc:2454] T 1bcaf6e3854a4fff9e8bfbcf1fd476eb P ff06107455ac4c638ba4dbd0061596d0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7c305125dc484bb0bcd6b6ccb48b8396 in term 1.
I20260812 05:20:53.340327 12186 leader_election.cc:302] T 1bcaf6e3854a4fff9e8bfbcf1fd476eb P 7c305125dc484bb0bcd6b6ccb48b8396 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7c305125dc484bb0bcd6b6ccb48b8396, ff06107455ac4c638ba4dbd0061596d0; no voters:
I20260812 05:20:53.340404 12969 raft_consensus.cc:2781] T 1bcaf6e3854a4fff9e8bfbcf1fd476eb P 7c305125dc484bb0bcd6b6ccb48b8396 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 05:20:53.340479 12969 raft_consensus.cc:686] T 1bcaf6e3854a4fff9e8bfbcf1fd476eb P 7c305125dc484bb0bcd6b6ccb48b8396 [term 1 LEADER]: Becoming Leader. State: Replica: 7c305125dc484bb0bcd6b6ccb48b8396, State: Running, Role: LEADER
I20260812 05:20:53.340533 12969 consensus_queue.cc:237] T 1bcaf6e3854a4fff9e8bfbcf1fd476eb P 7c305125dc484bb0bcd6b6ccb48b8396 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c305125dc484bb0bcd6b6ccb48b8396" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 46325 } } peers { permanent_uuid: "ff06107455ac4c638ba4dbd0061596d0" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 37165 } } peers { permanent_uuid: "3280a541e7e9497686edd61460573e5f" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 35461 } }
I20260812 05:20:53.340855 12920 tablet_replica.cc:439] T 1bcaf6e3854a4fff9e8bfbcf1fd476eb P 7c305125dc484bb0bcd6b6ccb48b8396: TxnStatusTablet state changed. Reason: New leader 7c305125dc484bb0bcd6b6ccb48b8396. Latest consensus state: current_term: 1 leader_uuid: "7c305125dc484bb0bcd6b6ccb48b8396" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c305125dc484bb0bcd6b6ccb48b8396" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 46325 } } peers { permanent_uuid: "ff06107455ac4c638ba4dbd0061596d0" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 37165 } } peers { permanent_uuid: "3280a541e7e9497686edd61460573e5f" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 35461 } } }
I20260812 05:20:53.340952 12920 tablet_replica.cc:442] T 1bcaf6e3854a4fff9e8bfbcf1fd476eb P 7c305125dc484bb0bcd6b6ccb48b8396: This TxnStatusTablet replica's current role is: LEADER
I20260812 05:20:53.341135 12031 catalog_manager.cc:5495] T 1bcaf6e3854a4fff9e8bfbcf1fd476eb P 7c305125dc484bb0bcd6b6ccb48b8396 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7c305125dc484bb0bcd6b6ccb48b8396 (127.31.100.1). New cstate: current_term: 1 leader_uuid: "7c305125dc484bb0bcd6b6ccb48b8396" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c305125dc484bb0bcd6b6ccb48b8396" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 46325 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ff06107455ac4c638ba4dbd0061596d0" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 37165 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3280a541e7e9497686edd61460573e5f" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 35461 } health_report { overall_health: UNKNOWN } } }
I20260812 05:20:53.341414 12980 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260812 05:20:53.341554 12980 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260812 05:20:53.350138 12962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:53.347021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.1:46325 (local address 127.0.0.1:42832)
0812 05:20:53.347131 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0812 05:20:53.347141 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0812 05:20:53.347185 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:20:53.347383 (+ 198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:20:53.347384 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:20:53.347385 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0812 05:20:53.347424 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:20:53.347426 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:20:53.348243 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:20:53.348244 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:20:53.349107 (+ 863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:20:53.349109 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:20:53.349607 (+ 498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:20:53.349608 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:20:53.349670 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:53.349686 (+ 16us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0812 05:20:53.349979 (+ 293us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0812 05:20:53.349987 (+ 8us) client_negotiation.cc:769] Sending connection context
0812 05:20:53.350021 (+ 34us) client_negotiation.cc:240] Negotiation successful
0812 05:20:53.350064 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":77}
I20260812 05:20:53.350376 12902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:53.347078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42832 (local address 127.31.100.1:46325)
0812 05:20:53.347254 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0812 05:20:53.347256 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 05:20:53.347262 (+ 6us) server_negotiation.cc:407] Connection header received
0812 05:20:53.347287 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:20:53.347288 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:20:53.347303 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:20:53.347349 (+ 46us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 05:20:53.347483 (+ 134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:53.348163 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:53.349195 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:53.349516 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:53.349568 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:53.349782 (+ 214us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 05:20:53.349899 (+ 117us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 05:20:53.349956 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0812 05:20:53.350294 (+ 338us) server_negotiation.cc:299] Negotiation successful
0812 05:20:53.350314 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160}
I20260812 05:20:53.351322 12546 raft_consensus.cc:1261] T 1bcaf6e3854a4fff9e8bfbcf1fd476eb P ff06107455ac4c638ba4dbd0061596d0 [term 1 FOLLOWER]: Refusing update from remote peer 7c305125dc484bb0bcd6b6ccb48b8396: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 05:20:53.351400 12781 raft_consensus.cc:1261] T 1bcaf6e3854a4fff9e8bfbcf1fd476eb P 3280a541e7e9497686edd61460573e5f [term 1 FOLLOWER]: Refusing update from remote peer 7c305125dc484bb0bcd6b6ccb48b8396: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 05:20:53.351558 12920 consensus_queue.cc:1035] T 1bcaf6e3854a4fff9e8bfbcf1fd476eb P 7c305125dc484bb0bcd6b6ccb48b8396 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff06107455ac4c638ba4dbd0061596d0" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 37165 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 05:20:53.351645 12969 consensus_queue.cc:1035] T 1bcaf6e3854a4fff9e8bfbcf1fd476eb P 7c305125dc484bb0bcd6b6ccb48b8396 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3280a541e7e9497686edd61460573e5f" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 35461 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 05:20:53.352300 12932 tablet_replica.cc:439] T 1bcaf6e3854a4fff9e8bfbcf1fd476eb P 3280a541e7e9497686edd61460573e5f: TxnStatusTablet state changed. Reason: New leader 7c305125dc484bb0bcd6b6ccb48b8396. Latest consensus state: current_term: 1 leader_uuid: "7c305125dc484bb0bcd6b6ccb48b8396" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c305125dc484bb0bcd6b6ccb48b8396" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 46325 } } peers { permanent_uuid: "ff06107455ac4c638ba4dbd0061596d0" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 37165 } } peers { permanent_uuid: "3280a541e7e9497686edd61460573e5f" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 35461 } } }
I20260812 05:20:53.352299 12915 tablet_replica.cc:439] T 1bcaf6e3854a4fff9e8bfbcf1fd476eb P ff06107455ac4c638ba4dbd0061596d0: TxnStatusTablet state changed. Reason: New leader 7c305125dc484bb0bcd6b6ccb48b8396. Latest consensus state: current_term: 1 leader_uuid: "7c305125dc484bb0bcd6b6ccb48b8396" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c305125dc484bb0bcd6b6ccb48b8396" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 46325 } } peers { permanent_uuid: "ff06107455ac4c638ba4dbd0061596d0" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 37165 } } peers { permanent_uuid: "3280a541e7e9497686edd61460573e5f" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 35461 } } }
I20260812 05:20:53.352385 12932 tablet_replica.cc:442] T 1bcaf6e3854a4fff9e8bfbcf1fd476eb P 3280a541e7e9497686edd61460573e5f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 05:20:53.352393 12915 tablet_replica.cc:442] T 1bcaf6e3854a4fff9e8bfbcf1fd476eb P ff06107455ac4c638ba4dbd0061596d0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 05:20:53.352867 12969 tablet_replica.cc:439] T 1bcaf6e3854a4fff9e8bfbcf1fd476eb P 7c305125dc484bb0bcd6b6ccb48b8396: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7c305125dc484bb0bcd6b6ccb48b8396" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c305125dc484bb0bcd6b6ccb48b8396" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 46325 } } peers { permanent_uuid: "ff06107455ac4c638ba4dbd0061596d0" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 37165 } } peers { permanent_uuid: "3280a541e7e9497686edd61460573e5f" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 35461 } } }
I20260812 05:20:53.352943 12969 tablet_replica.cc:442] T 1bcaf6e3854a4fff9e8bfbcf1fd476eb P 7c305125dc484bb0bcd6b6ccb48b8396: This TxnStatusTablet replica's current role is: LEADER
I20260812 05:20:53.353175 12915 tablet_replica.cc:439] T 1bcaf6e3854a4fff9e8bfbcf1fd476eb P ff06107455ac4c638ba4dbd0061596d0: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7c305125dc484bb0bcd6b6ccb48b8396" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c305125dc484bb0bcd6b6ccb48b8396" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 46325 } } peers { permanent_uuid: "ff06107455ac4c638ba4dbd0061596d0" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 37165 } } peers { permanent_uuid: "3280a541e7e9497686edd61460573e5f" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 35461 } } }
I20260812 05:20:53.353250 12915 tablet_replica.cc:442] T 1bcaf6e3854a4fff9e8bfbcf1fd476eb P ff06107455ac4c638ba4dbd0061596d0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 05:20:53.354429 12920 tablet_replica.cc:439] T 1bcaf6e3854a4fff9e8bfbcf1fd476eb P 7c305125dc484bb0bcd6b6ccb48b8396: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7c305125dc484bb0bcd6b6ccb48b8396" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c305125dc484bb0bcd6b6ccb48b8396" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 46325 } } peers { permanent_uuid: "ff06107455ac4c638ba4dbd0061596d0" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 37165 } } peers { permanent_uuid: "3280a541e7e9497686edd61460573e5f" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 35461 } } }
I20260812 05:20:53.354502 12920 tablet_replica.cc:442] T 1bcaf6e3854a4fff9e8bfbcf1fd476eb P 7c305125dc484bb0bcd6b6ccb48b8396: This TxnStatusTablet replica's current role is: LEADER
I20260812 05:20:53.355156 12932 tablet_replica.cc:439] T 1bcaf6e3854a4fff9e8bfbcf1fd476eb P 3280a541e7e9497686edd61460573e5f: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7c305125dc484bb0bcd6b6ccb48b8396" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c305125dc484bb0bcd6b6ccb48b8396" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 46325 } } peers { permanent_uuid: "ff06107455ac4c638ba4dbd0061596d0" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 37165 } } peers { permanent_uuid: "3280a541e7e9497686edd61460573e5f" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 35461 } } }
I20260812 05:20:53.355234 12932 tablet_replica.cc:442] T 1bcaf6e3854a4fff9e8bfbcf1fd476eb P 3280a541e7e9497686edd61460573e5f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 05:20:53.358372 12900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:53.355508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35252 (local address 127.31.100.2:37165)
0812 05:20:53.355546 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0812 05:20:53.355548 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 05:20:53.355555 (+ 7us) server_negotiation.cc:407] Connection header received
0812 05:20:53.355586 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:20:53.355587 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:20:53.355609 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:20:53.355671 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 05:20:53.355826 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:53.356541 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:53.357576 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:53.357871 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:53.357933 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:53.358065 (+ 132us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 05:20:53.358181 (+ 116us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 05:20:53.358249 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0812 05:20:53.358287 (+ 38us) server_negotiation.cc:299] Negotiation successful
0812 05:20:53.358306 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260812 05:20:53.363309 12635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:53.360564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.1:46325 (local address 127.31.100.2:39663)
0812 05:20:53.360625 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0812 05:20:53.360635 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0812 05:20:53.360677 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:20:53.360782 (+ 105us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:20:53.360782 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:20:53.360783 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0812 05:20:53.360842 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:20:53.360844 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:20:53.361735 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:20:53.361736 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:20:53.362481 (+ 745us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:20:53.362489 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:20:53.362938 (+ 449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:20:53.362939 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:20:53.363003 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:53.363019 (+ 16us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0812 05:20:53.363182 (+ 163us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0812 05:20:53.363184 (+ 2us) client_negotiation.cc:769] Sending connection context
0812 05:20:53.363214 (+ 30us) client_negotiation.cc:240] Negotiation successful
0812 05:20:53.363250 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260812 05:20:53.363525 12902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:53.360664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.100.2:39663 (local address 127.31.100.1:46325)
0812 05:20:53.360695 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0812 05:20:53.360697 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 05:20:53.360703 (+ 6us) server_negotiation.cc:407] Connection header received
0812 05:20:53.360723 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:20:53.360724 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:20:53.360732 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:20:53.360780 (+ 48us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 05:20:53.360916 (+ 136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:53.361647 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:53.362575 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:53.362827 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:53.362875 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:53.363104 (+ 229us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 05:20:53.363133 (+ 29us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 05:20:53.363165 (+ 32us) server_negotiation.cc:1035] Waiting for connection context
0812 05:20:53.363454 (+ 289us) server_negotiation.cc:299] Negotiation successful
0812 05:20:53.363473 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260812 05:20:53.365577 12781 raft_consensus.cc:1261] T ff9e2e977055461db36d37ff74022600 P 3280a541e7e9497686edd61460573e5f [term 1 FOLLOWER]: Refusing update from remote peer ff06107455ac4c638ba4dbd0061596d0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 05:20:53.365706 12303 raft_consensus.cc:1261] T ff9e2e977055461db36d37ff74022600 P 7c305125dc484bb0bcd6b6ccb48b8396 [term 1 FOLLOWER]: Refusing update from remote peer ff06107455ac4c638ba4dbd0061596d0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 05:20:53.365942 12951 consensus_queue.cc:1035] T ff9e2e977055461db36d37ff74022600 P ff06107455ac4c638ba4dbd0061596d0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3280a541e7e9497686edd61460573e5f" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 35461 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 05:20:53.366070 12915 consensus_queue.cc:1035] T ff9e2e977055461db36d37ff74022600 P ff06107455ac4c638ba4dbd0061596d0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7c305125dc484bb0bcd6b6ccb48b8396" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 46325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 05:20:53.373880 12900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:53.371317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.100.1:43313 (local address 127.31.100.2:37165)
0812 05:20:53.371362 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0812 05:20:53.371364 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 05:20:53.371455 (+ 91us) server_negotiation.cc:407] Connection header received
0812 05:20:53.371538 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:20:53.371539 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:20:53.371546 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:20:53.371579 (+ 33us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 05:20:53.371687 (+ 108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:53.372105 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:53.373190 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:53.373489 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:53.373550 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:53.373716 (+ 166us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 05:20:53.373739 (+ 23us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 05:20:53.373770 (+ 31us) server_negotiation.cc:1035] Waiting for connection context
0812 05:20:53.373822 (+ 52us) server_negotiation.cc:299] Negotiation successful
0812 05:20:53.373836 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260812 05:20:53.373968 12996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:53.371242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.2:37165 (local address 127.31.100.1:43313)
0812 05:20:53.371383 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0812 05:20:53.371394 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0812 05:20:53.371481 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:20:53.371583 (+ 102us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:20:53.371583 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:20:53.371584 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0812 05:20:53.371638 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:20:53.371639 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:20:53.372162 (+ 523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:20:53.372163 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:20:53.373120 (+ 957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:20:53.373122 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:20:53.373551 (+ 429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:20:53.373552 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:20:53.373625 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:53.373641 (+ 16us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0812 05:20:53.373773 (+ 132us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0812 05:20:53.373775 (+ 2us) client_negotiation.cc:769] Sending connection context
0812 05:20:53.373820 (+ 45us) client_negotiation.cc:240] Negotiation successful
0812 05:20:53.373858 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":83,"threads_started":1}
I20260812 05:20:53.383100 12030 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58030:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 05:20:53.383188 12030 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58030:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 05:20:53.383738 12030 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f895c32b0dae4fdba2c12bfabf74b1cf: Sending DeleteTablet for 3 replicas of tablet ff9e2e977055461db36d37ff74022600
I20260812 05:20:53.383975 12517 tablet_service.cc:1514] Processing DeleteTablet for tablet ff9e2e977055461db36d37ff74022600 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 05:20:53 UTC) from {username='oryx', principal='oryx/127.31.100.62@KRBTEST.COM'} at 127.0.0.1:35242
I20260812 05:20:53.384049 12271 tablet_service.cc:1514] Processing DeleteTablet for tablet ff9e2e977055461db36d37ff74022600 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 05:20:53 UTC) from {username='oryx', principal='oryx/127.31.100.62@KRBTEST.COM'} at 127.0.0.1:42818
I20260812 05:20:53.384083 12942 tablet_replica.cc:330] T ff9e2e977055461db36d37ff74022600 P ff06107455ac4c638ba4dbd0061596d0: stopping tablet replica
I20260812 05:20:53.384131 12943 tablet_replica.cc:330] T ff9e2e977055461db36d37ff74022600 P 7c305125dc484bb0bcd6b6ccb48b8396: stopping tablet replica
I20260812 05:20:53.384138 12942 raft_consensus.cc:2229] T ff9e2e977055461db36d37ff74022600 P ff06107455ac4c638ba4dbd0061596d0 [term 1 LEADER]: Raft consensus shutting down.
I20260812 05:20:53.384169 12943 raft_consensus.cc:2229] T ff9e2e977055461db36d37ff74022600 P 7c305125dc484bb0bcd6b6ccb48b8396 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 05:20:53.384197 12943 raft_consensus.cc:2258] T ff9e2e977055461db36d37ff74022600 P 7c305125dc484bb0bcd6b6ccb48b8396 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 05:20:53.384217 12942 raft_consensus.cc:2258] T ff9e2e977055461db36d37ff74022600 P ff06107455ac4c638ba4dbd0061596d0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 05:20:53.384290 12746 tablet_service.cc:1514] Processing DeleteTablet for tablet ff9e2e977055461db36d37ff74022600 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 05:20:53 UTC) from {username='oryx', principal='oryx/127.31.100.62@KRBTEST.COM'} at 127.0.0.1:41926
I20260812 05:20:53.384372 12940 tablet_replica.cc:330] T ff9e2e977055461db36d37ff74022600 P 3280a541e7e9497686edd61460573e5f: stopping tablet replica
I20260812 05:20:53.384423 12940 raft_consensus.cc:2229] T ff9e2e977055461db36d37ff74022600 P 3280a541e7e9497686edd61460573e5f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 05:20:53.384464 12940 raft_consensus.cc:2258] T ff9e2e977055461db36d37ff74022600 P 3280a541e7e9497686edd61460573e5f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 05:20:53.384757 12940 ts_tablet_manager.cc:1905] T ff9e2e977055461db36d37ff74022600 P 3280a541e7e9497686edd61460573e5f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 05:20:53.384757 12942 ts_tablet_manager.cc:1905] T ff9e2e977055461db36d37ff74022600 P ff06107455ac4c638ba4dbd0061596d0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 05:20:53.386301 12940 ts_tablet_manager.cc:1918] T ff9e2e977055461db36d37ff74022600 P 3280a541e7e9497686edd61460573e5f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260812 05:20:53.386301 12942 ts_tablet_manager.cc:1918] T ff9e2e977055461db36d37ff74022600 P ff06107455ac4c638ba4dbd0061596d0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260812 05:20:53.386353 12942 log.cc:1196] T ff9e2e977055461db36d37ff74022600 P ff06107455ac4c638ba4dbd0061596d0: Deleting WAL directory at /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786512017613366-32144-0/minicluster-data/ts-1/wal/wals/ff9e2e977055461db36d37ff74022600
I20260812 05:20:53.386353 12940 log.cc:1196] T ff9e2e977055461db36d37ff74022600 P 3280a541e7e9497686edd61460573e5f: Deleting WAL directory at /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786512017613366-32144-0/minicluster-data/ts-2/wal/wals/ff9e2e977055461db36d37ff74022600
I20260812 05:20:53.386551 12942 ts_tablet_manager.cc:1939] T ff9e2e977055461db36d37ff74022600 P ff06107455ac4c638ba4dbd0061596d0: Deleting consensus metadata
I20260812 05:20:53.386551 12940 ts_tablet_manager.cc:1939] T ff9e2e977055461db36d37ff74022600 P 3280a541e7e9497686edd61460573e5f: Deleting consensus metadata
I20260812 05:20:53.386927 12943 ts_tablet_manager.cc:1905] T ff9e2e977055461db36d37ff74022600 P 7c305125dc484bb0bcd6b6ccb48b8396: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 05:20:53.387027 12008 catalog_manager.cc:4841] TS ff06107455ac4c638ba4dbd0061596d0 (127.31.100.2:37165): tablet ff9e2e977055461db36d37ff74022600 (table test-table [id=fcb113180ad44f32b31f57c34c265163]) successfully deleted
I20260812 05:20:53.387311 12011 catalog_manager.cc:4841] TS 3280a541e7e9497686edd61460573e5f (127.31.100.3:35461): tablet ff9e2e977055461db36d37ff74022600 (table test-table [id=fcb113180ad44f32b31f57c34c265163]) successfully deleted
I20260812 05:20:53.388337 12943 ts_tablet_manager.cc:1918] T ff9e2e977055461db36d37ff74022600 P 7c305125dc484bb0bcd6b6ccb48b8396: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260812 05:20:53.388384 12943 log.cc:1196] T ff9e2e977055461db36d37ff74022600 P 7c305125dc484bb0bcd6b6ccb48b8396: Deleting WAL directory at /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786512017613366-32144-0/minicluster-data/ts-0/wal/wals/ff9e2e977055461db36d37ff74022600
I20260812 05:20:53.388573 12943 ts_tablet_manager.cc:1939] T ff9e2e977055461db36d37ff74022600 P 7c305125dc484bb0bcd6b6ccb48b8396: Deleting consensus metadata
I20260812 05:20:53.389027 12013 catalog_manager.cc:4841] TS 7c305125dc484bb0bcd6b6ccb48b8396 (127.31.100.1:46325): tablet ff9e2e977055461db36d37ff74022600 (table test-table [id=fcb113180ad44f32b31f57c34c265163]) successfully deleted
Aug 12 05:20:53 dist-test-slave-42z9 krb5kdc[11963](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512053, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 12 05:20:53 dist-test-slave-42z9 krb5kdc[11963](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.31.100.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 05:20:53 dist-test-slave-42z9 krb5kdc[11963](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.31.100.62@KRBTEST.COM, Server not found in Kerberos database
W20260812 05:20:53.417376 12118 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 05:20:53.412502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58040 (local address 127.31.100.62:39917)
0812 05:20:53.412550 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0812 05:20:53.412551 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0812 05:20:53.412680 (+ 129us) server_negotiation.cc:407] Connection header received
0812 05:20:53.412742 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:20:53.412743 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:20:53.412755 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:20:53.412798 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:20:53.413311 (+ 513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:53.413846 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:53.414559 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:53.414744 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:53.414797 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:53.417313 (+ 2516us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58040: BlockingRecv error: recv got EOF from 127.0.0.1:58040 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
Aug 12 05:20:53 dist-test-slave-42z9 krb5kdc[11963](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512053, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.31.100.62@KRBTEST.COM
I20260812 05:20:53.424393 12118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:53.418165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58054 (local address 127.31.100.62:39917)
0812 05:20:53.418199 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0812 05:20:53.418200 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0812 05:20:53.418376 (+ 176us) server_negotiation.cc:407] Connection header received
0812 05:20:53.418437 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:20:53.418438 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:20:53.418448 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:20:53.418488 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:20:53.418889 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:53.419398 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:53.420023 (+ 625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:53.420208 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:53.420258 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:53.422269 (+ 2011us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 05:20:53.422271 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 05:20:53.422271 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 05:20:53.422278 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 05:20:53.423471 (+ 1193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:20:53.423756 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:20:53.423757 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:20:53.423758 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:20:53.423811 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:20:53.423968 (+ 157us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:20:53.423969 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:20:53.423969 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:20:53.424070 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 05:20:53.424149 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0812 05:20:53.424241 (+ 92us) server_negotiation.cc:299] Negotiation successful
0812 05:20:53.424309 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260812 05:20:53.425729 12030 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58054:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 05:20:53.425881 12030 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 05:20:53.427196 12746 tablet_service.cc:1467] Processing CreateTablet for tablet 04d4fefc09da4318af7ee2d0bc517d10 (DEFAULT_TABLE table=test-table [id=5f09235ab5564f0bbc1f7bd364914a9e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 05:20:53.427213 12271 tablet_service.cc:1467] Processing CreateTablet for tablet 04d4fefc09da4318af7ee2d0bc517d10 (DEFAULT_TABLE table=test-table [id=5f09235ab5564f0bbc1f7bd364914a9e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 05:20:53.427325 12746 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 04d4fefc09da4318af7ee2d0bc517d10. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 05:20:53.427325 12271 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 04d4fefc09da4318af7ee2d0bc517d10. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 05:20:53.428587 12906 tablet_bootstrap.cc:492] T 04d4fefc09da4318af7ee2d0bc517d10 P 7c305125dc484bb0bcd6b6ccb48b8396: Bootstrap starting.
I20260812 05:20:53.428587 12905 tablet_bootstrap.cc:492] T 04d4fefc09da4318af7ee2d0bc517d10 P 3280a541e7e9497686edd61460573e5f: Bootstrap starting.
I20260812 05:20:53.428876 12905 tablet_bootstrap.cc:654] T 04d4fefc09da4318af7ee2d0bc517d10 P 3280a541e7e9497686edd61460573e5f: Neither blocks nor log segments found. Creating new log.
I20260812 05:20:53.429029 12517 tablet_service.cc:1467] Processing CreateTablet for tablet 04d4fefc09da4318af7ee2d0bc517d10 (DEFAULT_TABLE table=test-table [id=5f09235ab5564f0bbc1f7bd364914a9e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 05:20:53.429141 12517 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 04d4fefc09da4318af7ee2d0bc517d10. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 05:20:53.429943 12905 tablet_bootstrap.cc:492] T 04d4fefc09da4318af7ee2d0bc517d10 P 3280a541e7e9497686edd61460573e5f: No bootstrap required, opened a new log
I20260812 05:20:53.429994 12905 ts_tablet_manager.cc:1397] T 04d4fefc09da4318af7ee2d0bc517d10 P 3280a541e7e9497686edd61460573e5f: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260812 05:20:53.430114 12905 raft_consensus.cc:348] T 04d4fefc09da4318af7ee2d0bc517d10 P 3280a541e7e9497686edd61460573e5f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c305125dc484bb0bcd6b6ccb48b8396" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 46325 } } peers { permanent_uuid: "ff06107455ac4c638ba4dbd0061596d0" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 37165 } } peers { permanent_uuid: "3280a541e7e9497686edd61460573e5f" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 35461 } }
I20260812 05:20:53.430174 12905 raft_consensus.cc:374] T 04d4fefc09da4318af7ee2d0bc517d10 P 3280a541e7e9497686edd61460573e5f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 05:20:53.430190 12905 raft_consensus.cc:729] T 04d4fefc09da4318af7ee2d0bc517d10 P 3280a541e7e9497686edd61460573e5f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3280a541e7e9497686edd61460573e5f, State: Initialized, Role: FOLLOWER
I20260812 05:20:53.430230 12905 consensus_queue.cc:260] T 04d4fefc09da4318af7ee2d0bc517d10 P 3280a541e7e9497686edd61460573e5f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c305125dc484bb0bcd6b6ccb48b8396" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 46325 } } peers { permanent_uuid: "ff06107455ac4c638ba4dbd0061596d0" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 37165 } } peers { permanent_uuid: "3280a541e7e9497686edd61460573e5f" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 35461 } }
I20260812 05:20:53.430311 12905 ts_tablet_manager.cc:1428] T 04d4fefc09da4318af7ee2d0bc517d10 P 3280a541e7e9497686edd61460573e5f: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260812 05:20:53.430522 12907 tablet_bootstrap.cc:492] T 04d4fefc09da4318af7ee2d0bc517d10 P ff06107455ac4c638ba4dbd0061596d0: Bootstrap starting.
I20260812 05:20:53.430893 12907 tablet_bootstrap.cc:654] T 04d4fefc09da4318af7ee2d0bc517d10 P ff06107455ac4c638ba4dbd0061596d0: Neither blocks nor log segments found. Creating new log.
I20260812 05:20:53.431401 12906 tablet_bootstrap.cc:654] T 04d4fefc09da4318af7ee2d0bc517d10 P 7c305125dc484bb0bcd6b6ccb48b8396: Neither blocks nor log segments found. Creating new log.
I20260812 05:20:53.431805 12907 tablet_bootstrap.cc:492] T 04d4fefc09da4318af7ee2d0bc517d10 P ff06107455ac4c638ba4dbd0061596d0: No bootstrap required, opened a new log
I20260812 05:20:53.431847 12907 ts_tablet_manager.cc:1397] T 04d4fefc09da4318af7ee2d0bc517d10 P ff06107455ac4c638ba4dbd0061596d0: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260812 05:20:53.431982 12907 raft_consensus.cc:348] T 04d4fefc09da4318af7ee2d0bc517d10 P ff06107455ac4c638ba4dbd0061596d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c305125dc484bb0bcd6b6ccb48b8396" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 46325 } } peers { permanent_uuid: "ff06107455ac4c638ba4dbd0061596d0" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 37165 } } peers { permanent_uuid: "3280a541e7e9497686edd61460573e5f" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 35461 } }
I20260812 05:20:53.432035 12907 raft_consensus.cc:374] T 04d4fefc09da4318af7ee2d0bc517d10 P ff06107455ac4c638ba4dbd0061596d0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 05:20:53.432052 12907 raft_consensus.cc:729] T 04d4fefc09da4318af7ee2d0bc517d10 P ff06107455ac4c638ba4dbd0061596d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff06107455ac4c638ba4dbd0061596d0, State: Initialized, Role: FOLLOWER
I20260812 05:20:53.432088 12907 consensus_queue.cc:260] T 04d4fefc09da4318af7ee2d0bc517d10 P ff06107455ac4c638ba4dbd0061596d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c305125dc484bb0bcd6b6ccb48b8396" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 46325 } } peers { permanent_uuid: "ff06107455ac4c638ba4dbd0061596d0" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 37165 } } peers { permanent_uuid: "3280a541e7e9497686edd61460573e5f" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 35461 } }
I20260812 05:20:53.432178 12907 ts_tablet_manager.cc:1428] T 04d4fefc09da4318af7ee2d0bc517d10 P ff06107455ac4c638ba4dbd0061596d0: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260812 05:20:53.433913 12906 tablet_bootstrap.cc:492] T 04d4fefc09da4318af7ee2d0bc517d10 P 7c305125dc484bb0bcd6b6ccb48b8396: No bootstrap required, opened a new log
I20260812 05:20:53.433959 12906 ts_tablet_manager.cc:1397] T 04d4fefc09da4318af7ee2d0bc517d10 P 7c305125dc484bb0bcd6b6ccb48b8396: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20260812 05:20:53.434104 12906 raft_consensus.cc:348] T 04d4fefc09da4318af7ee2d0bc517d10 P 7c305125dc484bb0bcd6b6ccb48b8396 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c305125dc484bb0bcd6b6ccb48b8396" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 46325 } } peers { permanent_uuid: "ff06107455ac4c638ba4dbd0061596d0" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 37165 } } peers { permanent_uuid: "3280a541e7e9497686edd61460573e5f" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 35461 } }
I20260812 05:20:53.434165 12906 raft_consensus.cc:374] T 04d4fefc09da4318af7ee2d0bc517d10 P 7c305125dc484bb0bcd6b6ccb48b8396 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 05:20:53.434182 12906 raft_consensus.cc:729] T 04d4fefc09da4318af7ee2d0bc517d10 P 7c305125dc484bb0bcd6b6ccb48b8396 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c305125dc484bb0bcd6b6ccb48b8396, State: Initialized, Role: FOLLOWER
I20260812 05:20:53.434226 12906 consensus_queue.cc:260] T 04d4fefc09da4318af7ee2d0bc517d10 P 7c305125dc484bb0bcd6b6ccb48b8396 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c305125dc484bb0bcd6b6ccb48b8396" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 46325 } } peers { permanent_uuid: "ff06107455ac4c638ba4dbd0061596d0" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 37165 } } peers { permanent_uuid: "3280a541e7e9497686edd61460573e5f" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 35461 } }
I20260812 05:20:53.434314 12906 ts_tablet_manager.cc:1428] T 04d4fefc09da4318af7ee2d0bc517d10 P 7c305125dc484bb0bcd6b6ccb48b8396: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260812 05:20:53.668068 12932 raft_consensus.cc:482] T 04d4fefc09da4318af7ee2d0bc517d10 P 3280a541e7e9497686edd61460573e5f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 05:20:53.668177 12932 raft_consensus.cc:504] T 04d4fefc09da4318af7ee2d0bc517d10 P 3280a541e7e9497686edd61460573e5f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c305125dc484bb0bcd6b6ccb48b8396" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 46325 } } peers { permanent_uuid: "ff06107455ac4c638ba4dbd0061596d0" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 37165 } } peers { permanent_uuid: "3280a541e7e9497686edd61460573e5f" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 35461 } }
I20260812 05:20:53.668345 12932 leader_election.cc:288] T 04d4fefc09da4318af7ee2d0bc517d10 P 3280a541e7e9497686edd61460573e5f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7c305125dc484bb0bcd6b6ccb48b8396 (127.31.100.1:46325), ff06107455ac4c638ba4dbd0061596d0 (127.31.100.2:37165)
I20260812 05:20:53.668582 12546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04d4fefc09da4318af7ee2d0bc517d10" candidate_uuid: "3280a541e7e9497686edd61460573e5f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff06107455ac4c638ba4dbd0061596d0" is_pre_election: true
I20260812 05:20:53.668583 12303 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04d4fefc09da4318af7ee2d0bc517d10" candidate_uuid: "3280a541e7e9497686edd61460573e5f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c305125dc484bb0bcd6b6ccb48b8396" is_pre_election: true
I20260812 05:20:53.668679 12546 raft_consensus.cc:2454] T 04d4fefc09da4318af7ee2d0bc517d10 P ff06107455ac4c638ba4dbd0061596d0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3280a541e7e9497686edd61460573e5f in term 0.
I20260812 05:20:53.668684 12303 raft_consensus.cc:2454] T 04d4fefc09da4318af7ee2d0bc517d10 P 7c305125dc484bb0bcd6b6ccb48b8396 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3280a541e7e9497686edd61460573e5f in term 0.
I20260812 05:20:53.668824 12669 leader_election.cc:302] T 04d4fefc09da4318af7ee2d0bc517d10 P 3280a541e7e9497686edd61460573e5f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3280a541e7e9497686edd61460573e5f, ff06107455ac4c638ba4dbd0061596d0; no voters:
I20260812 05:20:53.668936 12932 raft_consensus.cc:2781] T 04d4fefc09da4318af7ee2d0bc517d10 P 3280a541e7e9497686edd61460573e5f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 05:20:53.668990 12932 raft_consensus.cc:482] T 04d4fefc09da4318af7ee2d0bc517d10 P 3280a541e7e9497686edd61460573e5f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 05:20:53.669013 12932 raft_consensus.cc:3037] T 04d4fefc09da4318af7ee2d0bc517d10 P 3280a541e7e9497686edd61460573e5f [term 0 FOLLOWER]: Advancing to term 1
I20260812 05:20:53.669828 12932 raft_consensus.cc:504] T 04d4fefc09da4318af7ee2d0bc517d10 P 3280a541e7e9497686edd61460573e5f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c305125dc484bb0bcd6b6ccb48b8396" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 46325 } } peers { permanent_uuid: "ff06107455ac4c638ba4dbd0061596d0" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 37165 } } peers { permanent_uuid: "3280a541e7e9497686edd61460573e5f" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 35461 } }
I20260812 05:20:53.669960 12932 leader_election.cc:288] T 04d4fefc09da4318af7ee2d0bc517d10 P 3280a541e7e9497686edd61460573e5f [CANDIDATE]: Term 1 election: Requested vote from peers 7c305125dc484bb0bcd6b6ccb48b8396 (127.31.100.1:46325), ff06107455ac4c638ba4dbd0061596d0 (127.31.100.2:37165)
I20260812 05:20:53.670125 12303 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04d4fefc09da4318af7ee2d0bc517d10" candidate_uuid: "3280a541e7e9497686edd61460573e5f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c305125dc484bb0bcd6b6ccb48b8396"
I20260812 05:20:53.670135 12546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04d4fefc09da4318af7ee2d0bc517d10" candidate_uuid: "3280a541e7e9497686edd61460573e5f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff06107455ac4c638ba4dbd0061596d0"
I20260812 05:20:53.670198 12546 raft_consensus.cc:3037] T 04d4fefc09da4318af7ee2d0bc517d10 P ff06107455ac4c638ba4dbd0061596d0 [term 0 FOLLOWER]: Advancing to term 1
I20260812 05:20:53.670200 12303 raft_consensus.cc:3037] T 04d4fefc09da4318af7ee2d0bc517d10 P 7c305125dc484bb0bcd6b6ccb48b8396 [term 0 FOLLOWER]: Advancing to term 1
I20260812 05:20:53.670828 12546 raft_consensus.cc:2454] T 04d4fefc09da4318af7ee2d0bc517d10 P ff06107455ac4c638ba4dbd0061596d0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3280a541e7e9497686edd61460573e5f in term 1.
I20260812 05:20:53.670866 12303 raft_consensus.cc:2454] T 04d4fefc09da4318af7ee2d0bc517d10 P 7c305125dc484bb0bcd6b6ccb48b8396 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3280a541e7e9497686edd61460573e5f in term 1.
I20260812 05:20:53.671026 12669 leader_election.cc:302] T 04d4fefc09da4318af7ee2d0bc517d10 P 3280a541e7e9497686edd61460573e5f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3280a541e7e9497686edd61460573e5f, ff06107455ac4c638ba4dbd0061596d0; no voters:
I20260812 05:20:53.671130 12932 raft_consensus.cc:2781] T 04d4fefc09da4318af7ee2d0bc517d10 P 3280a541e7e9497686edd61460573e5f [term 1 FOLLOWER]: Leader election won for term 1
I20260812 05:20:53.671181 12932 raft_consensus.cc:686] T 04d4fefc09da4318af7ee2d0bc517d10 P 3280a541e7e9497686edd61460573e5f [term 1 LEADER]: Becoming Leader. State: Replica: 3280a541e7e9497686edd61460573e5f, State: Running, Role: LEADER
I20260812 05:20:53.671233 12932 consensus_queue.cc:237] T 04d4fefc09da4318af7ee2d0bc517d10 P 3280a541e7e9497686edd61460573e5f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c305125dc484bb0bcd6b6ccb48b8396" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 46325 } } peers { permanent_uuid: "ff06107455ac4c638ba4dbd0061596d0" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 37165 } } peers { permanent_uuid: "3280a541e7e9497686edd61460573e5f" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 35461 } }
I20260812 05:20:53.671799 12030 catalog_manager.cc:5495] T 04d4fefc09da4318af7ee2d0bc517d10 P 3280a541e7e9497686edd61460573e5f reported cstate change: term changed from 0 to 1, leader changed from <none> to 3280a541e7e9497686edd61460573e5f (127.31.100.3). New cstate: current_term: 1 leader_uuid: "3280a541e7e9497686edd61460573e5f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c305125dc484bb0bcd6b6ccb48b8396" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 46325 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff06107455ac4c638ba4dbd0061596d0" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 37165 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3280a541e7e9497686edd61460573e5f" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 35461 } health_report { overall_health: HEALTHY } } }
I20260812 05:20:53.712210 12904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:53.710105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41944 (local address 127.31.100.3:35461)
0812 05:20:53.710158 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0812 05:20:53.710160 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 05:20:53.710181 (+ 21us) server_negotiation.cc:407] Connection header received
0812 05:20:53.710203 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:20:53.710204 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:20:53.710212 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:20:53.710248 (+ 36us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 05:20:53.710355 (+ 107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:53.710827 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:53.711496 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:53.711692 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:53.711755 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:53.711949 (+ 194us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 05:20:53.711987 (+ 38us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 05:20:53.712031 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0812 05:20:53.712113 (+ 82us) server_negotiation.cc:299] Negotiation successful
0812 05:20:53.712138 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260812 05:20:53.712994 12546 raft_consensus.cc:1261] T 04d4fefc09da4318af7ee2d0bc517d10 P ff06107455ac4c638ba4dbd0061596d0 [term 1 FOLLOWER]: Refusing update from remote peer 3280a541e7e9497686edd61460573e5f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 05:20:53.713011 12303 raft_consensus.cc:1261] T 04d4fefc09da4318af7ee2d0bc517d10 P 7c305125dc484bb0bcd6b6ccb48b8396 [term 1 FOLLOWER]: Refusing update from remote peer 3280a541e7e9497686edd61460573e5f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 05:20:53.713225 12932 consensus_queue.cc:1035] T 04d4fefc09da4318af7ee2d0bc517d10 P 3280a541e7e9497686edd61460573e5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "7c305125dc484bb0bcd6b6ccb48b8396" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 46325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 05:20:53.713402 12932 consensus_queue.cc:1035] T 04d4fefc09da4318af7ee2d0bc517d10 P 3280a541e7e9497686edd61460573e5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff06107455ac4c638ba4dbd0061596d0" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 37165 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 05:20:53.715816 12030 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58054:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 05:20:53.715893 12030 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58054:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 05:20:53.716486 12030 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f895c32b0dae4fdba2c12bfabf74b1cf: Sending DeleteTablet for 3 replicas of tablet 04d4fefc09da4318af7ee2d0bc517d10
I20260812 05:20:53.716717 12517 tablet_service.cc:1514] Processing DeleteTablet for tablet 04d4fefc09da4318af7ee2d0bc517d10 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 05:20:53 UTC) from {username='oryx', principal='oryx/127.31.100.62@KRBTEST.COM'} at 127.0.0.1:35242
I20260812 05:20:53.716766 12746 tablet_service.cc:1514] Processing DeleteTablet for tablet 04d4fefc09da4318af7ee2d0bc517d10 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 05:20:53 UTC) from {username='oryx', principal='oryx/127.31.100.62@KRBTEST.COM'} at 127.0.0.1:41926
I20260812 05:20:53.716825 12271 tablet_service.cc:1514] Processing DeleteTablet for tablet 04d4fefc09da4318af7ee2d0bc517d10 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 05:20:53 UTC) from {username='oryx', principal='oryx/127.31.100.62@KRBTEST.COM'} at 127.0.0.1:42818
I20260812 05:20:53.716849 12940 tablet_replica.cc:330] T 04d4fefc09da4318af7ee2d0bc517d10 P 3280a541e7e9497686edd61460573e5f: stopping tablet replica
I20260812 05:20:53.716909 12942 tablet_replica.cc:330] T 04d4fefc09da4318af7ee2d0bc517d10 P ff06107455ac4c638ba4dbd0061596d0: stopping tablet replica
I20260812 05:20:53.716917 12940 raft_consensus.cc:2229] T 04d4fefc09da4318af7ee2d0bc517d10 P 3280a541e7e9497686edd61460573e5f [term 1 LEADER]: Raft consensus shutting down.
I20260812 05:20:53.716960 12942 raft_consensus.cc:2229] T 04d4fefc09da4318af7ee2d0bc517d10 P ff06107455ac4c638ba4dbd0061596d0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 05:20:53.716971 12940 raft_consensus.cc:2258] T 04d4fefc09da4318af7ee2d0bc517d10 P 3280a541e7e9497686edd61460573e5f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 05:20:53.717011 12942 raft_consensus.cc:2258] T 04d4fefc09da4318af7ee2d0bc517d10 P ff06107455ac4c638ba4dbd0061596d0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 05:20:53.717082 12943 tablet_replica.cc:330] T 04d4fefc09da4318af7ee2d0bc517d10 P 7c305125dc484bb0bcd6b6ccb48b8396: stopping tablet replica
I20260812 05:20:53.717135 12943 raft_consensus.cc:2229] T 04d4fefc09da4318af7ee2d0bc517d10 P 7c305125dc484bb0bcd6b6ccb48b8396 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 05:20:53.717178 12943 raft_consensus.cc:2258] T 04d4fefc09da4318af7ee2d0bc517d10 P 7c305125dc484bb0bcd6b6ccb48b8396 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 05:20:53.717306 12940 ts_tablet_manager.cc:1905] T 04d4fefc09da4318af7ee2d0bc517d10 P 3280a541e7e9497686edd61460573e5f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 05:20:53.717454 12943 ts_tablet_manager.cc:1905] T 04d4fefc09da4318af7ee2d0bc517d10 P 7c305125dc484bb0bcd6b6ccb48b8396: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 05:20:53.718858 12940 ts_tablet_manager.cc:1918] T 04d4fefc09da4318af7ee2d0bc517d10 P 3280a541e7e9497686edd61460573e5f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 05:20:53.718792 12030 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58054:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260812 05:20:53.718865 12943 ts_tablet_manager.cc:1918] T 04d4fefc09da4318af7ee2d0bc517d10 P 7c305125dc484bb0bcd6b6ccb48b8396: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 05:20:53.718912 12943 log.cc:1196] T 04d4fefc09da4318af7ee2d0bc517d10 P 7c305125dc484bb0bcd6b6ccb48b8396: Deleting WAL directory at /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786512017613366-32144-0/minicluster-data/ts-0/wal/wals/04d4fefc09da4318af7ee2d0bc517d10
I20260812 05:20:53.718911 12940 log.cc:1196] T 04d4fefc09da4318af7ee2d0bc517d10 P 3280a541e7e9497686edd61460573e5f: Deleting WAL directory at /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786512017613366-32144-0/minicluster-data/ts-2/wal/wals/04d4fefc09da4318af7ee2d0bc517d10
W20260812 05:20:53.718948 12030 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 05:20:53.719125 12943 ts_tablet_manager.cc:1939] T 04d4fefc09da4318af7ee2d0bc517d10 P 7c305125dc484bb0bcd6b6ccb48b8396: Deleting consensus metadata
I20260812 05:20:53.719136 12940 ts_tablet_manager.cc:1939] T 04d4fefc09da4318af7ee2d0bc517d10 P 3280a541e7e9497686edd61460573e5f: Deleting consensus metadata
I20260812 05:20:53.719451 12011 catalog_manager.cc:4841] TS 3280a541e7e9497686edd61460573e5f (127.31.100.3:35461): tablet 04d4fefc09da4318af7ee2d0bc517d10 (table test-table [id=5f09235ab5564f0bbc1f7bd364914a9e]) successfully deleted
I20260812 05:20:53.719636 12942 ts_tablet_manager.cc:1905] T 04d4fefc09da4318af7ee2d0bc517d10 P ff06107455ac4c638ba4dbd0061596d0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 05:20:53.719470 12013 catalog_manager.cc:4841] TS 7c305125dc484bb0bcd6b6ccb48b8396 (127.31.100.1:46325): tablet 04d4fefc09da4318af7ee2d0bc517d10 (table test-table [id=5f09235ab5564f0bbc1f7bd364914a9e]) successfully deleted
I20260812 05:20:53.720790 12942 ts_tablet_manager.cc:1918] T 04d4fefc09da4318af7ee2d0bc517d10 P ff06107455ac4c638ba4dbd0061596d0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 05:20:53.720853 12942 log.cc:1196] T 04d4fefc09da4318af7ee2d0bc517d10 P ff06107455ac4c638ba4dbd0061596d0: Deleting WAL directory at /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786512017613366-32144-0/minicluster-data/ts-1/wal/wals/04d4fefc09da4318af7ee2d0bc517d10
I20260812 05:20:53.721006 12942 ts_tablet_manager.cc:1939] T 04d4fefc09da4318af7ee2d0bc517d10 P ff06107455ac4c638ba4dbd0061596d0: Deleting consensus metadata
I20260812 05:20:53.721341 12008 catalog_manager.cc:4841] TS ff06107455ac4c638ba4dbd0061596d0 (127.31.100.2:37165): tablet 04d4fefc09da4318af7ee2d0bc517d10 (table test-table [id=5f09235ab5564f0bbc1f7bd364914a9e]) successfully deleted
I20260812 05:20:53.721407 12517 tablet_service.cc:1467] Processing CreateTablet for tablet 784c0303c145427281855a91e1a2a81a (DEFAULT_TABLE table=test-table [id=95bee9edd817402dac666cbb83a8fc46]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 05:20:53.721426 12746 tablet_service.cc:1467] Processing CreateTablet for tablet 784c0303c145427281855a91e1a2a81a (DEFAULT_TABLE table=test-table [id=95bee9edd817402dac666cbb83a8fc46]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 05:20:53.721516 12271 tablet_service.cc:1467] Processing CreateTablet for tablet 784c0303c145427281855a91e1a2a81a (DEFAULT_TABLE table=test-table [id=95bee9edd817402dac666cbb83a8fc46]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 05:20:53.721539 12517 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 784c0303c145427281855a91e1a2a81a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 05:20:53.721544 12746 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 784c0303c145427281855a91e1a2a81a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 05:20:53.721602 12271 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 784c0303c145427281855a91e1a2a81a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 05:20:53.722417 12905 tablet_bootstrap.cc:492] T 784c0303c145427281855a91e1a2a81a P 3280a541e7e9497686edd61460573e5f: Bootstrap starting.
I20260812 05:20:53.722680 12906 tablet_bootstrap.cc:492] T 784c0303c145427281855a91e1a2a81a P 7c305125dc484bb0bcd6b6ccb48b8396: Bootstrap starting.
I20260812 05:20:53.722685 12907 tablet_bootstrap.cc:492] T 784c0303c145427281855a91e1a2a81a P ff06107455ac4c638ba4dbd0061596d0: Bootstrap starting.
I20260812 05:20:53.722752 12905 tablet_bootstrap.cc:654] T 784c0303c145427281855a91e1a2a81a P 3280a541e7e9497686edd61460573e5f: Neither blocks nor log segments found. Creating new log.
I20260812 05:20:53.722968 12906 tablet_bootstrap.cc:654] T 784c0303c145427281855a91e1a2a81a P 7c305125dc484bb0bcd6b6ccb48b8396: Neither blocks nor log segments found. Creating new log.
I20260812 05:20:53.722976 12907 tablet_bootstrap.cc:654] T 784c0303c145427281855a91e1a2a81a P ff06107455ac4c638ba4dbd0061596d0: Neither blocks nor log segments found. Creating new log.
I20260812 05:20:53.723281 12905 tablet_bootstrap.cc:492] T 784c0303c145427281855a91e1a2a81a P 3280a541e7e9497686edd61460573e5f: No bootstrap required, opened a new log
I20260812 05:20:53.723328 12905 ts_tablet_manager.cc:1397] T 784c0303c145427281855a91e1a2a81a P 3280a541e7e9497686edd61460573e5f: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260812 05:20:53.723445 12905 raft_consensus.cc:348] T 784c0303c145427281855a91e1a2a81a P 3280a541e7e9497686edd61460573e5f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c305125dc484bb0bcd6b6ccb48b8396" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 46325 } } peers { permanent_uuid: "ff06107455ac4c638ba4dbd0061596d0" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 37165 } } peers { permanent_uuid: "3280a541e7e9497686edd61460573e5f" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 35461 } }
I20260812 05:20:53.723497 12905 raft_consensus.cc:374] T 784c0303c145427281855a91e1a2a81a P 3280a541e7e9497686edd61460573e5f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 05:20:53.723512 12906 tablet_bootstrap.cc:492] T 784c0303c145427281855a91e1a2a81a P 7c305125dc484bb0bcd6b6ccb48b8396: No bootstrap required, opened a new log
I20260812 05:20:53.723523 12905 raft_consensus.cc:729] T 784c0303c145427281855a91e1a2a81a P 3280a541e7e9497686edd61460573e5f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3280a541e7e9497686edd61460573e5f, State: Initialized, Role: FOLLOWER
I20260812 05:20:53.723549 12906 ts_tablet_manager.cc:1397] T 784c0303c145427281855a91e1a2a81a P 7c305125dc484bb0bcd6b6ccb48b8396: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260812 05:20:53.723562 12905 consensus_queue.cc:260] T 784c0303c145427281855a91e1a2a81a P 3280a541e7e9497686edd61460573e5f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c305125dc484bb0bcd6b6ccb48b8396" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 46325 } } peers { permanent_uuid: "ff06107455ac4c638ba4dbd0061596d0" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 37165 } } peers { permanent_uuid: "3280a541e7e9497686edd61460573e5f" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 35461 } }
I20260812 05:20:53.723650 12905 ts_tablet_manager.cc:1428] T 784c0303c145427281855a91e1a2a81a P 3280a541e7e9497686edd61460573e5f: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260812 05:20:53.723634 12906 raft_consensus.cc:348] T 784c0303c145427281855a91e1a2a81a P 7c305125dc484bb0bcd6b6ccb48b8396 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c305125dc484bb0bcd6b6ccb48b8396" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 46325 } } peers { permanent_uuid: "ff06107455ac4c638ba4dbd0061596d0" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 37165 } } peers { permanent_uuid: "3280a541e7e9497686edd61460573e5f" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 35461 } }
I20260812 05:20:53.723688 12906 raft_consensus.cc:374] T 784c0303c145427281855a91e1a2a81a P 7c305125dc484bb0bcd6b6ccb48b8396 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 05:20:53.723706 12906 raft_consensus.cc:729] T 784c0303c145427281855a91e1a2a81a P 7c305125dc484bb0bcd6b6ccb48b8396 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c305125dc484bb0bcd6b6ccb48b8396, State: Initialized, Role: FOLLOWER
I20260812 05:20:53.723744 12906 consensus_queue.cc:260] T 784c0303c145427281855a91e1a2a81a P 7c305125dc484bb0bcd6b6ccb48b8396 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c305125dc484bb0bcd6b6ccb48b8396" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 46325 } } peers { permanent_uuid: "ff06107455ac4c638ba4dbd0061596d0" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 37165 } } peers { permanent_uuid: "3280a541e7e9497686edd61460573e5f" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 35461 } }
I20260812 05:20:53.723819 12906 ts_tablet_manager.cc:1428] T 784c0303c145427281855a91e1a2a81a P 7c305125dc484bb0bcd6b6ccb48b8396: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260812 05:20:53.723996 12907 tablet_bootstrap.cc:492] T 784c0303c145427281855a91e1a2a81a P ff06107455ac4c638ba4dbd0061596d0: No bootstrap required, opened a new log
I20260812 05:20:53.724045 12907 ts_tablet_manager.cc:1397] T 784c0303c145427281855a91e1a2a81a P ff06107455ac4c638ba4dbd0061596d0: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260812 05:20:53.724153 12907 raft_consensus.cc:348] T 784c0303c145427281855a91e1a2a81a P ff06107455ac4c638ba4dbd0061596d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c305125dc484bb0bcd6b6ccb48b8396" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 46325 } } peers { permanent_uuid: "ff06107455ac4c638ba4dbd0061596d0" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 37165 } } peers { permanent_uuid: "3280a541e7e9497686edd61460573e5f" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 35461 } }
I20260812 05:20:53.724213 12907 raft_consensus.cc:374] T 784c0303c145427281855a91e1a2a81a P ff06107455ac4c638ba4dbd0061596d0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 05:20:53.724231 12907 raft_consensus.cc:729] T 784c0303c145427281855a91e1a2a81a P ff06107455ac4c638ba4dbd0061596d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff06107455ac4c638ba4dbd0061596d0, State: Initialized, Role: FOLLOWER
I20260812 05:20:53.724270 12907 consensus_queue.cc:260] T 784c0303c145427281855a91e1a2a81a P ff06107455ac4c638ba4dbd0061596d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c305125dc484bb0bcd6b6ccb48b8396" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 46325 } } peers { permanent_uuid: "ff06107455ac4c638ba4dbd0061596d0" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 37165 } } peers { permanent_uuid: "3280a541e7e9497686edd61460573e5f" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 35461 } }
I20260812 05:20:53.724356 12907 ts_tablet_manager.cc:1428] T 784c0303c145427281855a91e1a2a81a P ff06107455ac4c638ba4dbd0061596d0: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260812 05:20:53.769793 12924 raft_consensus.cc:482] T 784c0303c145427281855a91e1a2a81a P 3280a541e7e9497686edd61460573e5f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 05:20:53.769886 12924 raft_consensus.cc:504] T 784c0303c145427281855a91e1a2a81a P 3280a541e7e9497686edd61460573e5f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c305125dc484bb0bcd6b6ccb48b8396" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 46325 } } peers { permanent_uuid: "ff06107455ac4c638ba4dbd0061596d0" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 37165 } } peers { permanent_uuid: "3280a541e7e9497686edd61460573e5f" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 35461 } }
I20260812 05:20:53.770031 12924 leader_election.cc:288] T 784c0303c145427281855a91e1a2a81a P 3280a541e7e9497686edd61460573e5f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7c305125dc484bb0bcd6b6ccb48b8396 (127.31.100.1:46325), ff06107455ac4c638ba4dbd0061596d0 (127.31.100.2:37165)
I20260812 05:20:53.770221 12303 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "784c0303c145427281855a91e1a2a81a" candidate_uuid: "3280a541e7e9497686edd61460573e5f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c305125dc484bb0bcd6b6ccb48b8396" is_pre_election: true
I20260812 05:20:53.770234 12546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "784c0303c145427281855a91e1a2a81a" candidate_uuid: "3280a541e7e9497686edd61460573e5f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff06107455ac4c638ba4dbd0061596d0" is_pre_election: true
I20260812 05:20:53.770313 12303 raft_consensus.cc:2454] T 784c0303c145427281855a91e1a2a81a P 7c305125dc484bb0bcd6b6ccb48b8396 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3280a541e7e9497686edd61460573e5f in term 0.
I20260812 05:20:53.770332 12546 raft_consensus.cc:2454] T 784c0303c145427281855a91e1a2a81a P ff06107455ac4c638ba4dbd0061596d0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3280a541e7e9497686edd61460573e5f in term 0.
I20260812 05:20:53.770458 12674 leader_election.cc:302] T 784c0303c145427281855a91e1a2a81a P 3280a541e7e9497686edd61460573e5f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3280a541e7e9497686edd61460573e5f, 7c305125dc484bb0bcd6b6ccb48b8396; no voters:
I20260812 05:20:53.770576 12924 raft_consensus.cc:2781] T 784c0303c145427281855a91e1a2a81a P 3280a541e7e9497686edd61460573e5f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 05:20:53.770627 12924 raft_consensus.cc:482] T 784c0303c145427281855a91e1a2a81a P 3280a541e7e9497686edd61460573e5f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 05:20:53.770695 12924 raft_consensus.cc:3037] T 784c0303c145427281855a91e1a2a81a P 3280a541e7e9497686edd61460573e5f [term 0 FOLLOWER]: Advancing to term 1
I20260812 05:20:53.771512 12924 raft_consensus.cc:504] T 784c0303c145427281855a91e1a2a81a P 3280a541e7e9497686edd61460573e5f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c305125dc484bb0bcd6b6ccb48b8396" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 46325 } } peers { permanent_uuid: "ff06107455ac4c638ba4dbd0061596d0" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 37165 } } peers { permanent_uuid: "3280a541e7e9497686edd61460573e5f" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 35461 } }
I20260812 05:20:53.771631 12924 leader_election.cc:288] T 784c0303c145427281855a91e1a2a81a P 3280a541e7e9497686edd61460573e5f [CANDIDATE]: Term 1 election: Requested vote from peers 7c305125dc484bb0bcd6b6ccb48b8396 (127.31.100.1:46325), ff06107455ac4c638ba4dbd0061596d0 (127.31.100.2:37165)
I20260812 05:20:53.771766 12303 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "784c0303c145427281855a91e1a2a81a" candidate_uuid: "3280a541e7e9497686edd61460573e5f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c305125dc484bb0bcd6b6ccb48b8396"
I20260812 05:20:53.771771 12546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "784c0303c145427281855a91e1a2a81a" candidate_uuid: "3280a541e7e9497686edd61460573e5f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff06107455ac4c638ba4dbd0061596d0"
I20260812 05:20:53.771843 12546 raft_consensus.cc:3037] T 784c0303c145427281855a91e1a2a81a P ff06107455ac4c638ba4dbd0061596d0 [term 0 FOLLOWER]: Advancing to term 1
I20260812 05:20:53.771845 12303 raft_consensus.cc:3037] T 784c0303c145427281855a91e1a2a81a P 7c305125dc484bb0bcd6b6ccb48b8396 [term 0 FOLLOWER]: Advancing to term 1
I20260812 05:20:53.772415 12546 raft_consensus.cc:2454] T 784c0303c145427281855a91e1a2a81a P ff06107455ac4c638ba4dbd0061596d0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3280a541e7e9497686edd61460573e5f in term 1.
I20260812 05:20:53.772425 12303 raft_consensus.cc:2454] T 784c0303c145427281855a91e1a2a81a P 7c305125dc484bb0bcd6b6ccb48b8396 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3280a541e7e9497686edd61460573e5f in term 1.
I20260812 05:20:53.772573 12674 leader_election.cc:302] T 784c0303c145427281855a91e1a2a81a P 3280a541e7e9497686edd61460573e5f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3280a541e7e9497686edd61460573e5f, 7c305125dc484bb0bcd6b6ccb48b8396; no voters:
I20260812 05:20:53.772673 12924 raft_consensus.cc:2781] T 784c0303c145427281855a91e1a2a81a P 3280a541e7e9497686edd61460573e5f [term 1 FOLLOWER]: Leader election won for term 1
I20260812 05:20:53.772732 12924 raft_consensus.cc:686] T 784c0303c145427281855a91e1a2a81a P 3280a541e7e9497686edd61460573e5f [term 1 LEADER]: Becoming Leader. State: Replica: 3280a541e7e9497686edd61460573e5f, State: Running, Role: LEADER
I20260812 05:20:53.772780 12924 consensus_queue.cc:237] T 784c0303c145427281855a91e1a2a81a P 3280a541e7e9497686edd61460573e5f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c305125dc484bb0bcd6b6ccb48b8396" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 46325 } } peers { permanent_uuid: "ff06107455ac4c638ba4dbd0061596d0" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 37165 } } peers { permanent_uuid: "3280a541e7e9497686edd61460573e5f" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 35461 } }
I20260812 05:20:53.773301 12031 catalog_manager.cc:5495] T 784c0303c145427281855a91e1a2a81a P 3280a541e7e9497686edd61460573e5f reported cstate change: term changed from 0 to 1, leader changed from <none> to 3280a541e7e9497686edd61460573e5f (127.31.100.3). New cstate: current_term: 1 leader_uuid: "3280a541e7e9497686edd61460573e5f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c305125dc484bb0bcd6b6ccb48b8396" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 46325 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff06107455ac4c638ba4dbd0061596d0" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 37165 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3280a541e7e9497686edd61460573e5f" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 35461 } health_report { overall_health: HEALTHY } } }
I20260812 05:20:53.786005 12902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:53.783991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.100.3:57995 (local address 127.31.100.1:46325)
0812 05:20:53.784054 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0812 05:20:53.784056 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 05:20:53.784140 (+ 84us) server_negotiation.cc:407] Connection header received
0812 05:20:53.784202 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:20:53.784202 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:20:53.784211 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:20:53.784245 (+ 34us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 05:20:53.784331 (+ 86us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:53.784781 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:53.785462 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:53.785667 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:53.785715 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:53.785845 (+ 130us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 05:20:53.785868 (+ 23us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 05:20:53.785898 (+ 30us) server_negotiation.cc:1035] Waiting for connection context
0812 05:20:53.785936 (+ 38us) server_negotiation.cc:299] Negotiation successful
0812 05:20:53.785952 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260812 05:20:53.786029 13043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:53.783924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.1:46325 (local address 127.31.100.3:57995)
0812 05:20:53.784088 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0812 05:20:53.784098 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0812 05:20:53.784152 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:20:53.784254 (+ 102us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:20:53.784254 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:20:53.784256 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0812 05:20:53.784298 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:20:53.784299 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:20:53.784825 (+ 526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:20:53.784826 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:20:53.785387 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:20:53.785389 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:20:53.785721 (+ 332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:20:53.785722 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:20:53.785774 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:53.785787 (+ 13us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0812 05:20:53.785904 (+ 117us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0812 05:20:53.785905 (+ 1us) client_negotiation.cc:769] Sending connection context
0812 05:20:53.785934 (+ 29us) client_negotiation.cc:240] Negotiation successful
0812 05:20:53.785974 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":104,"threads_started":1}
I20260812 05:20:53.787600 12303 raft_consensus.cc:1261] T 784c0303c145427281855a91e1a2a81a P 7c305125dc484bb0bcd6b6ccb48b8396 [term 1 FOLLOWER]: Refusing update from remote peer 3280a541e7e9497686edd61460573e5f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 05:20:53.787707 12546 raft_consensus.cc:1261] T 784c0303c145427281855a91e1a2a81a P ff06107455ac4c638ba4dbd0061596d0 [term 1 FOLLOWER]: Refusing update from remote peer 3280a541e7e9497686edd61460573e5f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 05:20:53.787775 12932 consensus_queue.cc:1035] T 784c0303c145427281855a91e1a2a81a P 3280a541e7e9497686edd61460573e5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "7c305125dc484bb0bcd6b6ccb48b8396" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 46325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 05:20:53.787876 12924 consensus_queue.cc:1035] T 784c0303c145427281855a91e1a2a81a P 3280a541e7e9497686edd61460573e5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff06107455ac4c638ba4dbd0061596d0" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 37165 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 05:20:53.794055 12904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:53.792057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.100.1:56873 (local address 127.31.100.3:35461)
0812 05:20:53.792094 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0812 05:20:53.792096 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 05:20:53.792104 (+ 8us) server_negotiation.cc:407] Connection header received
0812 05:20:53.792167 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:20:53.792167 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:20:53.792175 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:20:53.792233 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 05:20:53.792354 (+ 121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:53.792789 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:53.793483 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:53.793682 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:53.793734 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:53.793848 (+ 114us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 05:20:53.793880 (+ 32us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 05:20:53.793922 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0812 05:20:53.793971 (+ 49us) server_negotiation.cc:299] Negotiation successful
0812 05:20:53.793989 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260812 05:20:53.794068 12996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:53.792005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.3:35461 (local address 127.31.100.1:56873)
0812 05:20:53.792071 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0812 05:20:53.792082 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0812 05:20:53.792119 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:20:53.792262 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:20:53.792263 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:20:53.792264 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0812 05:20:53.792302 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:20:53.792303 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:20:53.792852 (+ 549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:20:53.792853 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:20:53.793397 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:20:53.793399 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:20:53.793739 (+ 340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:20:53.793740 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:20:53.793791 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:53.793797 (+ 6us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0812 05:20:53.793927 (+ 130us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0812 05:20:53.793928 (+ 1us) client_negotiation.cc:769] Sending connection context
0812 05:20:53.793969 (+ 41us) client_negotiation.cc:240] Negotiation successful
0812 05:20:53.794004 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20260812 05:20:53.799988 12031 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58054:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 05:20:53.800066 12031 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58054:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 05:20:53.800592 12031 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f895c32b0dae4fdba2c12bfabf74b1cf: Sending DeleteTablet for 3 replicas of tablet 784c0303c145427281855a91e1a2a81a
I20260812 05:20:53.800805 12271 tablet_service.cc:1514] Processing DeleteTablet for tablet 784c0303c145427281855a91e1a2a81a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 05:20:53 UTC) from {username='oryx', principal='oryx/127.31.100.62@KRBTEST.COM'} at 127.0.0.1:42818
I20260812 05:20:53.800841 12517 tablet_service.cc:1514] Processing DeleteTablet for tablet 784c0303c145427281855a91e1a2a81a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 05:20:53 UTC) from {username='oryx', principal='oryx/127.31.100.62@KRBTEST.COM'} at 127.0.0.1:35242
I20260812 05:20:53.800964 12943 tablet_replica.cc:330] T 784c0303c145427281855a91e1a2a81a P 7c305125dc484bb0bcd6b6ccb48b8396: stopping tablet replica
I20260812 05:20:53.801028 12943 raft_consensus.cc:2229] T 784c0303c145427281855a91e1a2a81a P 7c305125dc484bb0bcd6b6ccb48b8396 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 05:20:53.801067 12943 raft_consensus.cc:2258] T 784c0303c145427281855a91e1a2a81a P 7c305125dc484bb0bcd6b6ccb48b8396 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 05:20:53.801133 12942 tablet_replica.cc:330] T 784c0303c145427281855a91e1a2a81a P ff06107455ac4c638ba4dbd0061596d0: stopping tablet replica
I20260812 05:20:53.801122 32144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0waFvT/build/release/bin/kudu with pid 12152
I20260812 05:20:53.801184 12942 raft_consensus.cc:2229] T 784c0303c145427281855a91e1a2a81a P ff06107455ac4c638ba4dbd0061596d0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 05:20:53.801221 12942 raft_consensus.cc:2258] T 784c0303c145427281855a91e1a2a81a P ff06107455ac4c638ba4dbd0061596d0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 05:20:53.805560 12942 ts_tablet_manager.cc:1905] T 784c0303c145427281855a91e1a2a81a P ff06107455ac4c638ba4dbd0061596d0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 05:20:53.808741 12942 ts_tablet_manager.cc:1918] T 784c0303c145427281855a91e1a2a81a P ff06107455ac4c638ba4dbd0061596d0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260812 05:20:53.808830 12942 log.cc:1196] T 784c0303c145427281855a91e1a2a81a P ff06107455ac4c638ba4dbd0061596d0: Deleting WAL directory at /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786512017613366-32144-0/minicluster-data/ts-1/wal/wals/784c0303c145427281855a91e1a2a81a
I20260812 05:20:53.809098 12942 ts_tablet_manager.cc:1939] T 784c0303c145427281855a91e1a2a81a P ff06107455ac4c638ba4dbd0061596d0: Deleting consensus metadata
I20260812 05:20:53.809548 12008 catalog_manager.cc:4841] TS ff06107455ac4c638ba4dbd0061596d0 (127.31.100.2:37165): tablet 784c0303c145427281855a91e1a2a81a (table test-table [id=95bee9edd817402dac666cbb83a8fc46]) successfully deleted
I20260812 05:20:53.809983 12746 tablet_service.cc:1514] Processing DeleteTablet for tablet 784c0303c145427281855a91e1a2a81a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 05:20:53 UTC) from {username='oryx', principal='oryx/127.31.100.62@KRBTEST.COM'} at 127.0.0.1:41926
I20260812 05:20:53.810101 12940 tablet_replica.cc:330] T 784c0303c145427281855a91e1a2a81a P 3280a541e7e9497686edd61460573e5f: stopping tablet replica
I20260812 05:20:53.810176 12940 raft_consensus.cc:2229] T 784c0303c145427281855a91e1a2a81a P 3280a541e7e9497686edd61460573e5f [term 1 LEADER]: Raft consensus shutting down.
I20260812 05:20:53.810245 12940 raft_consensus.cc:2258] T 784c0303c145427281855a91e1a2a81a P 3280a541e7e9497686edd61460573e5f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 05:20:53.810511 12940 ts_tablet_manager.cc:1905] T 784c0303c145427281855a91e1a2a81a P 3280a541e7e9497686edd61460573e5f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 05:20:53.811640 12940 ts_tablet_manager.cc:1918] T 784c0303c145427281855a91e1a2a81a P 3280a541e7e9497686edd61460573e5f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260812 05:20:53.811691 12940 log.cc:1196] T 784c0303c145427281855a91e1a2a81a P 3280a541e7e9497686edd61460573e5f: Deleting WAL directory at /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786512017613366-32144-0/minicluster-data/ts-2/wal/wals/784c0303c145427281855a91e1a2a81a
I20260812 05:20:53.811887 12940 ts_tablet_manager.cc:1939] T 784c0303c145427281855a91e1a2a81a P 3280a541e7e9497686edd61460573e5f: Deleting consensus metadata
I20260812 05:20:53.812192 12011 catalog_manager.cc:4841] TS 3280a541e7e9497686edd61460573e5f (127.31.100.3:35461): tablet 784c0303c145427281855a91e1a2a81a (table test-table [id=95bee9edd817402dac666cbb83a8fc46]) successfully deleted
W20260812 05:20:53.818329 12013 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.31.100.1:46325 (error 108)
I20260812 05:20:53.818763 13049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:53.818620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.1:46325 (local address 127.0.0.1:42838)
0812 05:20:53.818694 (+ 74us) negotiation.cc:107] Waiting for socket to connect
0812 05:20:53.818729 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.100.1:46325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60,"thread_start_us":35,"threads_started":1}
W20260812 05:20:53.818874 12013 catalog_manager.cc:4584] TS 7c305125dc484bb0bcd6b6ccb48b8396 (127.31.100.1:46325): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 784c0303c145427281855a91e1a2a81a: Network error: Client connection negotiation failed: client connection to 127.31.100.1:46325: connect: Connection refused (error 111)
I20260812 05:20:53.818957 32144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0waFvT/build/release/bin/kudu with pid 12412
I20260812 05:20:53.826674 32144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0waFvT/build/release/bin/kudu with pid 12638
I20260812 05:20:53.837013 32144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0waFvT/build/release/bin/kudu with pid 11982
2026-08-12T05:20:53Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (2753 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786512017613366-32144-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 05:20:53 dist-test-slave-42z9 krb5kdc[13053](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 05:20:53 dist-test-slave-42z9 krb5kdc[13053](info): set up 2 sockets
Aug 12 05:20:53 dist-test-slave-42z9 krb5kdc[13053](info): commencing operation
krb5kdc: starting...
W20260812 05:20:55.005714 32144 mini_kdc.cc:121] Time spent starting KDC: real 1.143s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786512017613366-32144-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786512017613366-32144-0/krb5kdc/test-user.keytab.
Aug 12 05:20:55 dist-test-slave-42z9 krb5kdc[13053](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512055, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T05:20:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T05:20:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.100.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.100.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.100.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.62.keytab.
Entry for principal kudu/127.31.100.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.62.keytab.
I20260812 05:20:55.122468 32144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0waFvT/build/release/bin/kudu
/tmp/dist-test-task0waFvT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786512017613366-32144-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786512017613366-32144-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786512017613366-32144-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786512017613366-32144-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.100.62:33987
--webserver_interface=127.31.100.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.62.keytab
--principal=kudu/127.31.100.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.100.20:45595
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.100.62:33987
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786512017613366-32144-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786512017613366-32144-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786512017613366-32144-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 05:20:55.204638 13074 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 05:20:55.204834 13074 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 05:20:55.204855 13074 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 05:20:55.205976 13074 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 05:20:55.206015 13074 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 05:20:55.206029 13074 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 05:20:55.206040 13074 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 05:20:55.206050 13074 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 05:20:55.207501 13074 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.100.20:45595
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786512017613366-32144-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786512017613366-32144-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.100.62:33987
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.100.62:33987
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786512017613366-32144-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.100.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13074
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786512017613366-32144-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 04:20:34 UTC on ccycloud.quasar-phqdqp.root.comops.site
build id 4581871
I20260812 05:20:55.207743 13074 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 05:20:55.207952 13074 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 05:20:55.212595 13084 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 05:20:55.212594 13086 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 05:20:55.212625 13074 server_base.cc:1034] running on GCE node
W20260812 05:20:55.212621 13090 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 05:20:55.213032 13074 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 05:20:55.213290 13074 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 05:20:55.214429 13074 hybrid_clock.cc:648] HybridClock initialized: now 1786512055214392 us; error 50 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-08-12T05:20:55Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1376 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786512017613366-32144-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 05:20:55 dist-test-slave-42z9 krb5kdc[13098](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 05:20:55 dist-test-slave-42z9 krb5kdc[13098](info): set up 2 sockets
Aug 12 05:20:55 dist-test-slave-42z9 krb5kdc[13098](info): commencing operation
krb5kdc: starting...
W20260812 05:20:56.364539 32144 mini_kdc.cc:121] Time spent starting KDC: real 1.125s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786512017613366-32144-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786512017613366-32144-0/krb5kdc/test-user.keytab.
Aug 12 05:20:56 dist-test-slave-42z9 krb5kdc[13098](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512056, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T05:20:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T05:20:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.100.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.100.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.100.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.62.keytab.
Entry for principal kudu/127.31.100.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.62.keytab.
I20260812 05:20:56.476145 32144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0waFvT/build/release/bin/kudu
/tmp/dist-test-task0waFvT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786512017613366-32144-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786512017613366-32144-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786512017613366-32144-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786512017613366-32144-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.100.62:41525
--webserver_interface=127.31.100.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.62.keytab
--principal=kudu/127.31.100.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.100.20:35097
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.100.62:41525
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786512017613366-32144-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786512017613366-32144-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786512017613366-32144-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 05:20:56.555660 13120 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 05:20:56.555846 13120 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 05:20:56.555866 13120 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 05:20:56.556979 13120 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 05:20:56.557019 13120 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 05:20:56.557032 13120 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 05:20:56.557055 13120 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 05:20:56.557077 13120 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 05:20:56.558583 13120 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.100.20:35097
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786512017613366-32144-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786512017613366-32144-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.100.62:41525
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.62.keytab
--principal=kudu/127.31.100.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.100.62:41525
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786512017613366-32144-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.100.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13120
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786512017613366-32144-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 04:20:34 UTC on ccycloud.quasar-phqdqp.root.comops.site
build id 4581871
I20260812 05:20:56.558805 13120 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 05:20:56.559005 13120 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 05:20:56.563710 13127 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 05:20:56.563704 13128 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 05:20:56.563753 13120 server_base.cc:1034] running on GCE node
W20260812 05:20:56.563714 13131 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 05:20:56.564152 13120 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 05:20:56.564466 13120 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 05:20:56.565609 13120 hybrid_clock.cc:648] HybridClock initialized: now 1786512056565593 us; error 28 us; skew 500 ppm
Aug 12 05:20:56 dist-test-slave-42z9 krb5kdc[13098](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512056, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.100.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 05:20:56.572103 13120 init.cc:377] Logged in from keytab as kudu/127.31.100.62@KRBTEST.COM (short username kudu)
I20260812 05:20:56.572532 13120 webserver.cc:460] Webserver started at http://127.31.100.62:45581/ using document root <none> and password file <none>
I20260812 05:20:56.572750 13120 fs_manager.cc:359] Metadata directory not provided
I20260812 05:20:56.572798 13120 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 05:20:56.572922 13120 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 05:20:56.574036 13120 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786512017613366-32144-0/minicluster-data/master-0/data/instance:
uuid: "a2275c27dce54c69bb2d43da953129a0"
format_stamp: "Formatted at 2026-08-12 05:20:56 on dist-test-slave-42z9"
server_key: "e24070c15848290852328acb1c198f4e"
server_key_iv: "5c79d26745005fb3012dcc3c3b1614d2"
server_key_version: "encryptionkey@0"
I20260812 05:20:56.574369 13120 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786512017613366-32144-0/minicluster-data/master-0/wal/instance:
uuid: "a2275c27dce54c69bb2d43da953129a0"
format_stamp: "Formatted at 2026-08-12 05:20:56 on dist-test-slave-42z9"
server_key: "e24070c15848290852328acb1c198f4e"
server_key_iv: "5c79d26745005fb3012dcc3c3b1614d2"
server_key_version: "encryptionkey@0"
I20260812 05:20:56.575572 13120 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260812 05:20:56.576292 13145 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 05:20:56.576449 13120 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260812 05:20:56.576543 13120 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786512017613366-32144-0/minicluster-data/master-0/data,/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786512017613366-32144-0/minicluster-data/master-0/wal
uuid: "a2275c27dce54c69bb2d43da953129a0"
format_stamp: "Formatted at 2026-08-12 05:20:56 on dist-test-slave-42z9"
server_key: "e24070c15848290852328acb1c198f4e"
server_key_iv: "5c79d26745005fb3012dcc3c3b1614d2"
server_key_version: "encryptionkey@0"
I20260812 05:20:56.576608 13120 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786512017613366-32144-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786512017613366-32144-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786512017613366-32144-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 05:20:56.606431 13120 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 05:20:56.608207 13120 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 05:20:56.608338 13120 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 05:20:56.612270 13120 rpc_server.cc:307] RPC server started. Bound to: 127.31.100.62:41525
I20260812 05:20:56.612301 13226 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.100.62:41525 every 8 connection(s)
I20260812 05:20:56.612749 13120 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786512017613366-32144-0/minicluster-data/master-0/data/info.pb
I20260812 05:20:56.613399 13228 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 05:20:56.615628 13228 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a2275c27dce54c69bb2d43da953129a0: Bootstrap starting.
I20260812 05:20:56.616174 13228 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a2275c27dce54c69bb2d43da953129a0: Neither blocks nor log segments found. Creating new log.
I20260812 05:20:56.616403 13228 log.cc:824] T 00000000000000000000000000000000 P a2275c27dce54c69bb2d43da953129a0: Log is configured to *not* fsync() on all Append() calls
I20260812 05:20:56.617033 13228 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a2275c27dce54c69bb2d43da953129a0: No bootstrap required, opened a new log
I20260812 05:20:56.618604 13228 raft_consensus.cc:348] T 00000000000000000000000000000000 P a2275c27dce54c69bb2d43da953129a0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2275c27dce54c69bb2d43da953129a0" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 41525 } }
I20260812 05:20:56.618739 13228 raft_consensus.cc:374] T 00000000000000000000000000000000 P a2275c27dce54c69bb2d43da953129a0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 05:20:56.618760 13228 raft_consensus.cc:729] T 00000000000000000000000000000000 P a2275c27dce54c69bb2d43da953129a0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2275c27dce54c69bb2d43da953129a0, State: Initialized, Role: FOLLOWER
I20260812 05:20:56.618865 13228 consensus_queue.cc:260] T 00000000000000000000000000000000 P a2275c27dce54c69bb2d43da953129a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2275c27dce54c69bb2d43da953129a0" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 41525 } }
I20260812 05:20:56.618959 13228 raft_consensus.cc:388] T 00000000000000000000000000000000 P a2275c27dce54c69bb2d43da953129a0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 05:20:56.618983 13228 raft_consensus.cc:482] T 00000000000000000000000000000000 P a2275c27dce54c69bb2d43da953129a0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 05:20:56.619027 13228 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a2275c27dce54c69bb2d43da953129a0 [term 0 FOLLOWER]: Advancing to term 1
I20260812 05:20:56.619884 13228 raft_consensus.cc:504] T 00000000000000000000000000000000 P a2275c27dce54c69bb2d43da953129a0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2275c27dce54c69bb2d43da953129a0" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 41525 } }
I20260812 05:20:56.619973 13228 leader_election.cc:302] T 00000000000000000000000000000000 P a2275c27dce54c69bb2d43da953129a0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a2275c27dce54c69bb2d43da953129a0; no voters:
I20260812 05:20:56.620097 13228 leader_election.cc:288] T 00000000000000000000000000000000 P a2275c27dce54c69bb2d43da953129a0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 05:20:56.620143 13233 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a2275c27dce54c69bb2d43da953129a0 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 05:20:56.620317 13233 raft_consensus.cc:686] T 00000000000000000000000000000000 P a2275c27dce54c69bb2d43da953129a0 [term 1 LEADER]: Becoming Leader. State: Replica: a2275c27dce54c69bb2d43da953129a0, State: Running, Role: LEADER
I20260812 05:20:56.620327 13228 sys_catalog.cc:564] T 00000000000000000000000000000000 P a2275c27dce54c69bb2d43da953129a0 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 05:20:56.620415 13233 consensus_queue.cc:237] T 00000000000000000000000000000000 P a2275c27dce54c69bb2d43da953129a0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2275c27dce54c69bb2d43da953129a0" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 41525 } }
I20260812 05:20:56.620762 32144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0waFvT/build/release/bin/kudu as pid 13120
I20260812 05:20:56.620875 32144 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786512017613366-32144-0/minicluster-data/master-0/wal/instance
I20260812 05:20:56.621070 32144 external_mini_cluster.cc:1428] Setting key c86a5aeb726203227818a0e13633a564
I20260812 05:20:56.621469 13234 sys_catalog.cc:455] T 00000000000000000000000000000000 P a2275c27dce54c69bb2d43da953129a0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a2275c27dce54c69bb2d43da953129a0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2275c27dce54c69bb2d43da953129a0" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 41525 } } }
I20260812 05:20:56.621568 13234 sys_catalog.cc:458] T 00000000000000000000000000000000 P a2275c27dce54c69bb2d43da953129a0 [sys.catalog]: This master's current role is: LEADER
I20260812 05:20:56.621765 13235 sys_catalog.cc:455] T 00000000000000000000000000000000 P a2275c27dce54c69bb2d43da953129a0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a2275c27dce54c69bb2d43da953129a0. Latest consensus state: current_term: 1 leader_uuid: "a2275c27dce54c69bb2d43da953129a0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2275c27dce54c69bb2d43da953129a0" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 41525 } } }
I20260812 05:20:56.621974 13235 sys_catalog.cc:458] T 00000000000000000000000000000000 P a2275c27dce54c69bb2d43da953129a0 [sys.catalog]: This master's current role is: LEADER
W20260812 05:20:56.622154 13255 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a2275c27dce54c69bb2d43da953129a0: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 05:20:56.622210 13255 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 05:20:56.622257 13257 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 05:20:56.622412 13257 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 05:20:56.623804 13257 catalog_manager.cc:1347] Generated new cluster ID: 75d8d0f990874adb8785d6c6d015c114
I20260812 05:20:56.623862 13257 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 12 05:20:56 dist-test-slave-42z9 krb5kdc[13098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512056, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.100.62@KRBTEST.COM
I20260812 05:20:56.632280 13260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:56.621697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56656 (local address 127.31.100.62:41525)
0812 05:20:56.621940 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0812 05:20:56.621950 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 05:20:56.622191 (+ 241us) server_negotiation.cc:407] Connection header received
0812 05:20:56.622304 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:20:56.622311 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:20:56.622366 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:20:56.622488 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:20:56.623129 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:56.626026 (+ 2897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:56.626747 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:56.627153 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:56.627246 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:56.629342 (+ 2096us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 05:20:56.629346 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 05:20:56.629346 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 05:20:56.629357 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0812 05:20:56.630849 (+ 1492us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:20:56.631171 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:20:56.631174 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:20:56.631174 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:20:56.631274 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:20:56.631452 (+ 178us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:20:56.631453 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:20:56.631453 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:20:56.631671 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 05:20:56.631818 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0812 05:20:56.631905 (+ 87us) server_negotiation.cc:299] Negotiation successful
0812 05:20:56.632128 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":175,"thread_start_us":66,"threads_started":1}
I20260812 05:20:56.638453 13257 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 05:20:56.638928 13257 catalog_manager.cc:1504] Loading token signing keys...
I20260812 05:20:56.643267 13257 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a2275c27dce54c69bb2d43da953129a0: Generated new TSK 0
I20260812 05:20:56.643431 13257 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.31.100.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.100.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.100.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.1.keytab.
Entry for principal kudu/127.31.100.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.1.keytab.
I20260812 05:20:56.668649 32144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0waFvT/build/release/bin/kudu
/tmp/dist-test-task0waFvT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786512017613366-32144-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786512017613366-32144-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786512017613366-32144-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786512017613366-32144-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.100.1:0
--local_ip_for_outbound_sockets=127.31.100.1
--webserver_interface=127.31.100.1
--webserver_port=0
--tserver_master_addrs=127.31.100.62:41525
--keytab_file=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.1.keytab
--principal=kudu/127.31.100.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.100.20:35097
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786512017613366-32144-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786512017613366-32144-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786512017613366-32144-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 05:20:56.752269 13271 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 05:20:56.752480 13271 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 05:20:56.752504 13271 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 05:20:56.753814 13271 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 05:20:56.753873 13271 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 05:20:56.753916 13271 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.100.1
I20260812 05:20:56.755489 13271 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.100.20:35097
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786512017613366-32144-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786512017613366-32144-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.1.keytab
--principal=kudu/127.31.100.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.100.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786512017613366-32144-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.100.1
--webserver_port=0
--tserver_master_addrs=127.31.100.62:41525
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13271
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.100.1
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786512017613366-32144-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 04:20:34 UTC on ccycloud.quasar-phqdqp.root.comops.site
build id 4581871
I20260812 05:20:56.755717 13271 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 05:20:56.755920 13271 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 05:20:56.760602 13277 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 05:20:56.760609 13276 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 05:20:56.760809 13271 server_base.cc:1034] running on GCE node
W20260812 05:20:56.760622 13280 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 05:20:56.761173 13271 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 05:20:56.761399 13271 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 05:20:56.762609 13271 hybrid_clock.cc:648] HybridClock initialized: now 1786512056762585 us; error 39 us; skew 500 ppm
Aug 12 05:20:56 dist-test-slave-42z9 krb5kdc[13098](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512056, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.100.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 05:20:56.765221 13271 init.cc:377] Logged in from keytab as kudu/127.31.100.1@KRBTEST.COM (short username kudu)
I20260812 05:20:56.765674 13271 webserver.cc:460] Webserver started at http://127.31.100.1:42253/ using document root <none> and password file <none>
I20260812 05:20:56.765869 13271 fs_manager.cc:359] Metadata directory not provided
I20260812 05:20:56.765914 13271 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 05:20:56.766037 13271 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 05:20:56.766971 13271 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786512017613366-32144-0/minicluster-data/ts-0/data/instance:
uuid: "3e68316b411745f69a83096255039bb1"
format_stamp: "Formatted at 2026-08-12 05:20:56 on dist-test-slave-42z9"
server_key: "31e3953ba354e405de805cb2e172153e"
server_key_iv: "a0c90013f506a5c6f2d59f37d88ce213"
server_key_version: "encryptionkey@0"
I20260812 05:20:56.767287 13271 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786512017613366-32144-0/minicluster-data/ts-0/wal/instance:
uuid: "3e68316b411745f69a83096255039bb1"
format_stamp: "Formatted at 2026-08-12 05:20:56 on dist-test-slave-42z9"
server_key: "31e3953ba354e405de805cb2e172153e"
server_key_iv: "a0c90013f506a5c6f2d59f37d88ce213"
server_key_version: "encryptionkey@0"
I20260812 05:20:56.768528 13271 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260812 05:20:56.769135 13294 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 05:20:56.769295 13271 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260812 05:20:56.769378 13271 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786512017613366-32144-0/minicluster-data/ts-0/data,/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786512017613366-32144-0/minicluster-data/ts-0/wal
uuid: "3e68316b411745f69a83096255039bb1"
format_stamp: "Formatted at 2026-08-12 05:20:56 on dist-test-slave-42z9"
server_key: "31e3953ba354e405de805cb2e172153e"
server_key_iv: "a0c90013f506a5c6f2d59f37d88ce213"
server_key_version: "encryptionkey@0"
I20260812 05:20:56.769443 13271 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786512017613366-32144-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786512017613366-32144-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786512017613366-32144-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 05:20:56.796707 13271 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 05:20:56.798691 13271 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 05:20:56.798820 13271 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 05:20:56.799024 13271 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 05:20:56.799295 13271 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 05:20:56.799328 13271 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 05:20:56.799353 13271 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 05:20:56.799383 13271 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 05:20:56.804639 13271 rpc_server.cc:307] RPC server started. Bound to: 127.31.100.1:43473
I20260812 05:20:56.804699 13490 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.100.1:43473 every 8 connection(s)
I20260812 05:20:56.804955 13271 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786512017613366-32144-0/minicluster-data/ts-0/data/info.pb
Aug 12 05:20:56 dist-test-slave-42z9 krb5kdc[13098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512056, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.100.1@KRBTEST.COM for kudu/127.31.100.62@KRBTEST.COM
I20260812 05:20:56.813092 13260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:56.805606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.100.1:46091 (local address 127.31.100.62:41525)
0812 05:20:56.805674 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0812 05:20:56.805676 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 05:20:56.805855 (+ 179us) server_negotiation.cc:407] Connection header received
0812 05:20:56.806305 (+ 450us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:20:56.806306 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:20:56.806318 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:20:56.806364 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:20:56.807254 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:56.807751 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:56.808684 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:56.808868 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:56.808914 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:56.810785 (+ 1871us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 05:20:56.810787 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 05:20:56.810788 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 05:20:56.810794 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0812 05:20:56.811928 (+ 1134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:20:56.812311 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:20:56.812312 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:20:56.812312 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:20:56.812365 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:20:56.812563 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:20:56.812564 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:20:56.812564 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:20:56.812659 (+ 95us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 05:20:56.812705 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0812 05:20:56.812901 (+ 196us) server_negotiation.cc:299] Negotiation successful
0812 05:20:56.813003 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260812 05:20:56.813122 32144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0waFvT/build/release/bin/kudu as pid 13271
I20260812 05:20:56.813141 13494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:56.805583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.62:41525 (local address 127.31.100.1:46091)
0812 05:20:56.805732 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0812 05:20:56.805767 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0812 05:20:56.806226 (+ 459us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:20:56.806454 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:20:56.806457 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:20:56.806988 (+ 531us) client_negotiation.cc:189] Negotiated authn=SASL
0812 05:20:56.807173 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:20:56.807178 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:20:56.807832 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:20:56.807833 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:20:56.808618 (+ 785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:20:56.808620 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:20:56.808939 (+ 319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:20:56.808940 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:20:56.809040 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:56.809430 (+ 390us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 05:20:56.809452 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0812 05:20:56.810705 (+ 1253us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 05:20:56.812029 (+ 1324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 05:20:56.812036 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 05:20:56.812038 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 05:20:56.812255 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 05:20:56.812418 (+ 163us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 05:20:56.812419 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 05:20:56.812419 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 05:20:56.812513 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 05:20:56.812703 (+ 190us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 05:20:56.812706 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 05:20:56.812805 (+ 99us) client_negotiation.cc:769] Sending connection context
0812 05:20:56.812880 (+ 75us) client_negotiation.cc:240] Negotiation successful
0812 05:20:56.812991 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":82,"threads_started":1}
I20260812 05:20:56.813213 32144 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786512017613366-32144-0/minicluster-data/ts-0/wal/instance
I20260812 05:20:56.813373 32144 external_mini_cluster.cc:1428] Setting key 1bc9bf11897ece2ff4aa7698cb583f14
I20260812 05:20:56.813918 13491 heartbeater.cc:347] Connected to a master server at 127.31.100.62:41525
I20260812 05:20:56.814018 13491 heartbeater.cc:464] Registering TS with master...
I20260812 05:20:56.814181 13491 heartbeater.cc:511] Master 127.31.100.62:41525 requested a full tablet report, sending...
I20260812 05:20:56.814680 13171 ts_manager.cc:194] Registered new tserver with Master: 3e68316b411745f69a83096255039bb1 (127.31.100.1:43473)
I20260812 05:20:56.815223 13171 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.100.1@KRBTEST.COM'} at 127.31.100.1:46091
WARNING: no policy specified for kudu/127.31.100.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.100.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.100.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.2.keytab.
Entry for principal kudu/127.31.100.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.2.keytab.
I20260812 05:20:56.836064 32144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0waFvT/build/release/bin/kudu
/tmp/dist-test-task0waFvT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786512017613366-32144-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786512017613366-32144-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786512017613366-32144-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786512017613366-32144-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.100.2:0
--local_ip_for_outbound_sockets=127.31.100.2
--webserver_interface=127.31.100.2
--webserver_port=0
--tserver_master_addrs=127.31.100.62:41525
--keytab_file=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.2.keytab
--principal=kudu/127.31.100.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.100.20:35097
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786512017613366-32144-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786512017613366-32144-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786512017613366-32144-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 05:20:56.914628 13499 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 05:20:56.914808 13499 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 05:20:56.914829 13499 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 05:20:56.915947 13499 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 05:20:56.915987 13499 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 05:20:56.916013 13499 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.100.2
I20260812 05:20:56.917414 13499 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.100.20:35097
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786512017613366-32144-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786512017613366-32144-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.2.keytab
--principal=kudu/127.31.100.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.100.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786512017613366-32144-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.100.2
--webserver_port=0
--tserver_master_addrs=127.31.100.62:41525
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13499
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.100.2
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786512017613366-32144-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 04:20:34 UTC on ccycloud.quasar-phqdqp.root.comops.site
build id 4581871
I20260812 05:20:56.917646 13499 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 05:20:56.917843 13499 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 05:20:56.922638 13505 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 05:20:56.922678 13511 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 05:20:56.922777 13499 server_base.cc:1034] running on GCE node
W20260812 05:20:56.922644 13507 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 05:20:56.923048 13499 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 05:20:56.923236 13499 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 05:20:56.924368 13499 hybrid_clock.cc:648] HybridClock initialized: now 1786512056924360 us; error 26 us; skew 500 ppm
Aug 12 05:20:56 dist-test-slave-42z9 krb5kdc[13098](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512056, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.100.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 05:20:56.926674 13499 init.cc:377] Logged in from keytab as kudu/127.31.100.2@KRBTEST.COM (short username kudu)
I20260812 05:20:56.927088 13499 webserver.cc:460] Webserver started at http://127.31.100.2:38829/ using document root <none> and password file <none>
I20260812 05:20:56.927276 13499 fs_manager.cc:359] Metadata directory not provided
I20260812 05:20:56.927315 13499 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 05:20:56.927420 13499 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 05:20:56.928333 13499 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786512017613366-32144-0/minicluster-data/ts-1/data/instance:
uuid: "7e63bace79bd4e3e81b406d0efe31082"
format_stamp: "Formatted at 2026-08-12 05:20:56 on dist-test-slave-42z9"
server_key: "ca8758c68d1cf5f0c443ef856117722d"
server_key_iv: "de15eb999882a609b66e1fec3020dc60"
server_key_version: "encryptionkey@0"
I20260812 05:20:56.928608 13499 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786512017613366-32144-0/minicluster-data/ts-1/wal/instance:
uuid: "7e63bace79bd4e3e81b406d0efe31082"
format_stamp: "Formatted at 2026-08-12 05:20:56 on dist-test-slave-42z9"
server_key: "ca8758c68d1cf5f0c443ef856117722d"
server_key_iv: "de15eb999882a609b66e1fec3020dc60"
server_key_version: "encryptionkey@0"
I20260812 05:20:56.929777 13499 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260812 05:20:56.930425 13523 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 05:20:56.930593 13499 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260812 05:20:56.930672 13499 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786512017613366-32144-0/minicluster-data/ts-1/data,/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786512017613366-32144-0/minicluster-data/ts-1/wal
uuid: "7e63bace79bd4e3e81b406d0efe31082"
format_stamp: "Formatted at 2026-08-12 05:20:56 on dist-test-slave-42z9"
server_key: "ca8758c68d1cf5f0c443ef856117722d"
server_key_iv: "de15eb999882a609b66e1fec3020dc60"
server_key_version: "encryptionkey@0"
I20260812 05:20:56.930722 13499 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786512017613366-32144-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786512017613366-32144-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786512017613366-32144-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 05:20:56.936954 13499 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 05:20:56.938833 13499 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 05:20:56.938935 13499 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 05:20:56.939095 13499 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 05:20:56.939313 13499 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 05:20:56.939342 13499 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 05:20:56.939393 13499 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 05:20:56.939419 13499 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 05:20:56.944659 13499 rpc_server.cc:307] RPC server started. Bound to: 127.31.100.2:35847
I20260812 05:20:56.944721 13727 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.100.2:35847 every 8 connection(s)
I20260812 05:20:56.944955 13499 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786512017613366-32144-0/minicluster-data/ts-1/data/info.pb
I20260812 05:20:56.950099 32144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0waFvT/build/release/bin/kudu as pid 13499
I20260812 05:20:56.950192 32144 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786512017613366-32144-0/minicluster-data/ts-1/wal/instance
I20260812 05:20:56.950353 32144 external_mini_cluster.cc:1428] Setting key e0ad72eca736dfdaee69c5af4b3d5807
Aug 12 05:20:56 dist-test-slave-42z9 krb5kdc[13098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512056, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.100.2@KRBTEST.COM for kudu/127.31.100.62@KRBTEST.COM
I20260812 05:20:56.953310 13260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:56.945784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.100.2:42799 (local address 127.31.100.62:41525)
0812 05:20:56.945846 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0812 05:20:56.945848 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 05:20:56.946012 (+ 164us) server_negotiation.cc:407] Connection header received
0812 05:20:56.946422 (+ 410us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:20:56.946424 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:20:56.946434 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:20:56.946474 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:20:56.947308 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:56.947801 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:56.948702 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:56.948876 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:56.948917 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:56.950835 (+ 1918us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 05:20:56.950837 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 05:20:56.950837 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 05:20:56.950843 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0812 05:20:56.952089 (+ 1246us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:20:56.952454 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:20:56.952456 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:20:56.952456 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:20:56.952515 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:20:56.952731 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:20:56.952732 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:20:56.952732 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:20:56.952839 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 05:20:56.952892 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0812 05:20:56.953139 (+ 247us) server_negotiation.cc:299] Negotiation successful
0812 05:20:56.953223 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260812 05:20:56.953464 13731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:56.945748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.62:41525 (local address 127.31.100.2:42799)
0812 05:20:56.945907 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0812 05:20:56.945941 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0812 05:20:56.946354 (+ 413us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:20:56.946525 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:20:56.946528 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:20:56.947037 (+ 509us) client_negotiation.cc:189] Negotiated authn=SASL
0812 05:20:56.947215 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:20:56.947219 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:20:56.947868 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:20:56.947868 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:20:56.948642 (+ 774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:20:56.948645 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:20:56.948926 (+ 281us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:20:56.948926 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:20:56.949021 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:56.949375 (+ 354us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 05:20:56.949383 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0812 05:20:56.950756 (+ 1373us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 05:20:56.952167 (+ 1411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 05:20:56.952174 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 05:20:56.952176 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 05:20:56.952383 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 05:20:56.952570 (+ 187us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 05:20:56.952571 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 05:20:56.952571 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 05:20:56.952675 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 05:20:56.952893 (+ 218us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 05:20:56.952900 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 05:20:56.953025 (+ 125us) client_negotiation.cc:769] Sending connection context
0812 05:20:56.953109 (+ 84us) client_negotiation.cc:240] Negotiation successful
0812 05:20:56.953250 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":83,"threads_started":1}
I20260812 05:20:56.953855 13728 heartbeater.cc:347] Connected to a master server at 127.31.100.62:41525
I20260812 05:20:56.953936 13728 heartbeater.cc:464] Registering TS with master...
I20260812 05:20:56.954097 13728 heartbeater.cc:511] Master 127.31.100.62:41525 requested a full tablet report, sending...
I20260812 05:20:56.954442 13171 ts_manager.cc:194] Registered new tserver with Master: 7e63bace79bd4e3e81b406d0efe31082 (127.31.100.2:35847)
I20260812 05:20:56.954942 13171 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.100.2@KRBTEST.COM'} at 127.31.100.2:42799
WARNING: no policy specified for kudu/127.31.100.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.100.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.100.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.3.keytab.
Entry for principal kudu/127.31.100.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.3.keytab.
I20260812 05:20:56.973958 32144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0waFvT/build/release/bin/kudu
/tmp/dist-test-task0waFvT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786512017613366-32144-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786512017613366-32144-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786512017613366-32144-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786512017613366-32144-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.100.3:0
--local_ip_for_outbound_sockets=127.31.100.3
--webserver_interface=127.31.100.3
--webserver_port=0
--tserver_master_addrs=127.31.100.62:41525
--keytab_file=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.3.keytab
--principal=kudu/127.31.100.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.100.20:35097
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786512017613366-32144-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786512017613366-32144-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786512017613366-32144-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 05:20:57.052932 13736 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 05:20:57.053112 13736 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 05:20:57.053133 13736 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 05:20:57.054222 13736 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 05:20:57.054255 13736 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 05:20:57.054279 13736 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.100.3
I20260812 05:20:57.055626 13736 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.100.20:35097
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786512017613366-32144-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786512017613366-32144-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.3.keytab
--principal=kudu/127.31.100.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.100.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786512017613366-32144-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.100.3
--webserver_port=0
--tserver_master_addrs=127.31.100.62:41525
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13736
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.100.3
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786512017613366-32144-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 04:20:34 UTC on ccycloud.quasar-phqdqp.root.comops.site
build id 4581871
I20260812 05:20:57.055790 13736 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 05:20:57.055956 13736 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 05:20:57.060704 13754 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 05:20:57.060706 13749 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 05:20:57.060714 13748 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 05:20:57.060833 13736 server_base.cc:1034] running on GCE node
I20260812 05:20:57.061091 13736 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 05:20:57.061307 13736 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 05:20:57.062450 13736 hybrid_clock.cc:648] HybridClock initialized: now 1786512057062436 us; error 31 us; skew 500 ppm
Aug 12 05:20:57 dist-test-slave-42z9 krb5kdc[13098](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512057, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.100.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 05:20:57.064761 13736 init.cc:377] Logged in from keytab as kudu/127.31.100.3@KRBTEST.COM (short username kudu)
I20260812 05:20:57.065201 13736 webserver.cc:460] Webserver started at http://127.31.100.3:38369/ using document root <none> and password file <none>
I20260812 05:20:57.065416 13736 fs_manager.cc:359] Metadata directory not provided
I20260812 05:20:57.065510 13736 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 05:20:57.065639 13736 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 05:20:57.066653 13736 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786512017613366-32144-0/minicluster-data/ts-2/data/instance:
uuid: "7e98f49f4a8345b9a9b24bbf55623d3e"
format_stamp: "Formatted at 2026-08-12 05:20:57 on dist-test-slave-42z9"
server_key: "56c4adb935a0a71162e0f2901359ccf2"
server_key_iv: "8362a45092d8684e1e6670fafb65ad20"
server_key_version: "encryptionkey@0"
I20260812 05:20:57.066983 13736 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786512017613366-32144-0/minicluster-data/ts-2/wal/instance:
uuid: "7e98f49f4a8345b9a9b24bbf55623d3e"
format_stamp: "Formatted at 2026-08-12 05:20:57 on dist-test-slave-42z9"
server_key: "56c4adb935a0a71162e0f2901359ccf2"
server_key_iv: "8362a45092d8684e1e6670fafb65ad20"
server_key_version: "encryptionkey@0"
I20260812 05:20:57.068183 13736 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260812 05:20:57.068897 13765 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 05:20:57.069060 13736 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260812 05:20:57.069145 13736 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786512017613366-32144-0/minicluster-data/ts-2/data,/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786512017613366-32144-0/minicluster-data/ts-2/wal
uuid: "7e98f49f4a8345b9a9b24bbf55623d3e"
format_stamp: "Formatted at 2026-08-12 05:20:57 on dist-test-slave-42z9"
server_key: "56c4adb935a0a71162e0f2901359ccf2"
server_key_iv: "8362a45092d8684e1e6670fafb65ad20"
server_key_version: "encryptionkey@0"
I20260812 05:20:57.069209 13736 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786512017613366-32144-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786512017613366-32144-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786512017613366-32144-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 05:20:57.076995 13736 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 05:20:57.078786 13736 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 05:20:57.078943 13736 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 05:20:57.079131 13736 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 05:20:57.079414 13736 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 05:20:57.079452 13736 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 05:20:57.079478 13736 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 05:20:57.079493 13736 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 05:20:57.084878 13736 rpc_server.cc:307] RPC server started. Bound to: 127.31.100.3:36375
I20260812 05:20:57.084944 13951 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.100.3:36375 every 8 connection(s)
I20260812 05:20:57.085258 13736 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786512017613366-32144-0/minicluster-data/ts-2/data/info.pb
I20260812 05:20:57.087904 32144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0waFvT/build/release/bin/kudu as pid 13736
I20260812 05:20:57.088007 32144 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786512017613366-32144-0/minicluster-data/ts-2/wal/instance
I20260812 05:20:57.088171 32144 external_mini_cluster.cc:1428] Setting key 7cee87931f8a8d3b48cad8ba3973e6d8
Aug 12 05:20:57 dist-test-slave-42z9 krb5kdc[13098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512057, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.100.3@KRBTEST.COM for kudu/127.31.100.62@KRBTEST.COM
I20260812 05:20:57.093780 13260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:57.085918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.100.3:51557 (local address 127.31.100.62:41525)
0812 05:20:57.085973 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0812 05:20:57.085975 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 05:20:57.086162 (+ 187us) server_negotiation.cc:407] Connection header received
0812 05:20:57.086590 (+ 428us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:20:57.086592 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:20:57.086601 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:20:57.086658 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:20:57.087492 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:57.088029 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:57.089220 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:57.089413 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:57.089508 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:57.091446 (+ 1938us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 05:20:57.091447 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 05:20:57.091448 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 05:20:57.091456 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0812 05:20:57.092524 (+ 1068us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:20:57.092877 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:20:57.092878 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:20:57.092879 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:20:57.092939 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:20:57.093153 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:20:57.093153 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:20:57.093154 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:20:57.093256 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 05:20:57.093310 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0812 05:20:57.093615 (+ 305us) server_negotiation.cc:299] Negotiation successful
0812 05:20:57.093693 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260812 05:20:57.093941 13956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:57.085882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.62:41525 (local address 127.31.100.3:51557)
0812 05:20:57.086032 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0812 05:20:57.086079 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0812 05:20:57.086515 (+ 436us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:20:57.086712 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:20:57.086715 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:20:57.087248 (+ 533us) client_negotiation.cc:189] Negotiated authn=SASL
0812 05:20:57.087424 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:20:57.087428 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:20:57.088110 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:20:57.088111 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:20:57.089144 (+ 1033us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:20:57.089147 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:20:57.089523 (+ 376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:20:57.089524 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:20:57.089649 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:57.090045 (+ 396us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 05:20:57.090054 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0812 05:20:57.091357 (+ 1303us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 05:20:57.092598 (+ 1241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 05:20:57.092605 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 05:20:57.092607 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 05:20:57.092811 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 05:20:57.092996 (+ 185us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 05:20:57.092997 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 05:20:57.092997 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 05:20:57.093096 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 05:20:57.093312 (+ 216us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 05:20:57.093318 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 05:20:57.093483 (+ 165us) client_negotiation.cc:769] Sending connection context
0812 05:20:57.093583 (+ 100us) client_negotiation.cc:240] Negotiation successful
0812 05:20:57.093742 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":68,"threads_started":1}
I20260812 05:20:57.094336 13953 heartbeater.cc:347] Connected to a master server at 127.31.100.62:41525
I20260812 05:20:57.094422 13953 heartbeater.cc:464] Registering TS with master...
I20260812 05:20:57.094584 13953 heartbeater.cc:511] Master 127.31.100.62:41525 requested a full tablet report, sending...
I20260812 05:20:57.094993 13171 ts_manager.cc:194] Registered new tserver with Master: 7e98f49f4a8345b9a9b24bbf55623d3e (127.31.100.3:36375)
I20260812 05:20:57.095346 13171 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.100.3@KRBTEST.COM'} at 127.31.100.3:51557
I20260812 05:20:57.099769 32144 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.31.100.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.31.100.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.31.100.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786512017613366-32144-0/krb5kdc/oryx_127.31.100.1.keytab.
Entry for principal oryx/127.31.100.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786512017613366-32144-0/krb5kdc/oryx_127.31.100.1.keytab.
WARNING: no policy specified for oryx/127.31.100.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.31.100.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.31.100.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786512017613366-32144-0/krb5kdc/oryx_127.31.100.2.keytab.
Entry for principal oryx/127.31.100.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786512017613366-32144-0/krb5kdc/oryx_127.31.100.2.keytab.
WARNING: no policy specified for oryx/127.31.100.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.31.100.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.31.100.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786512017613366-32144-0/krb5kdc/oryx_127.31.100.3.keytab.
Entry for principal oryx/127.31.100.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786512017613366-32144-0/krb5kdc/oryx_127.31.100.3.keytab.
I20260812 05:20:57.167668 32144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0waFvT/build/release/bin/kudu with pid 13271
I20260812 05:20:57.177959 32144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0waFvT/build/release/bin/kudu with pid 13499
I20260812 05:20:57.190476 32144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0waFvT/build/release/bin/kudu with pid 13736
I20260812 05:20:57.202701 32144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0waFvT/build/release/bin/kudu with pid 13120
I20260812 05:20:57.213272 32144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0waFvT/build/release/bin/kudu
/tmp/dist-test-task0waFvT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786512017613366-32144-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786512017613366-32144-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786512017613366-32144-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786512017613366-32144-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.100.62:41525
--webserver_interface=127.31.100.62
--webserver_port=45581
--keytab_file=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.62.keytab
--principal=kudu/127.31.100.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.100.20:35097
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.100.62:41525
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786512017613366-32144-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786512017613366-32144-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786512017613366-32144-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 05:20:57.292624 13968 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 05:20:57.292820 13968 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 05:20:57.292843 13968 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 05:20:57.293980 13968 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 05:20:57.294021 13968 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 05:20:57.294034 13968 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 05:20:57.294044 13968 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 05:20:57.294055 13968 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 05:20:57.295508 13968 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.100.20:35097
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786512017613366-32144-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786512017613366-32144-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.100.62:41525
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.62.keytab
--principal=kudu/127.31.100.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.100.62:41525
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786512017613366-32144-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.100.62
--webserver_port=45581
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13968
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786512017613366-32144-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 04:20:34 UTC on ccycloud.quasar-phqdqp.root.comops.site
build id 4581871
I20260812 05:20:57.295720 13968 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 05:20:57.295915 13968 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 05:20:57.300485 13987 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 05:20:57.300542 13975 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 05:20:57.300485 13976 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 05:20:57.300518 13968 server_base.cc:1034] running on GCE node
I20260812 05:20:57.300861 13968 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 05:20:57.301131 13968 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 05:20:57.302296 13968 hybrid_clock.cc:648] HybridClock initialized: now 1786512057302280 us; error 29 us; skew 500 ppm
Aug 12 05:20:57 dist-test-slave-42z9 krb5kdc[13098](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512057, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.100.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 05:20:57.304633 13968 init.cc:377] Logged in from keytab as kudu/127.31.100.62@KRBTEST.COM (short username kudu)
I20260812 05:20:57.305015 13968 webserver.cc:460] Webserver started at http://127.31.100.62:45581/ using document root <none> and password file <none>
I20260812 05:20:57.305222 13968 fs_manager.cc:359] Metadata directory not provided
I20260812 05:20:57.305284 13968 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 05:20:57.306532 13968 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260812 05:20:57.307086 14000 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 05:20:57.307230 13968 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260812 05:20:57.307312 13968 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786512017613366-32144-0/minicluster-data/master-0/data,/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786512017613366-32144-0/minicluster-data/master-0/wal
uuid: "a2275c27dce54c69bb2d43da953129a0"
format_stamp: "Formatted at 2026-08-12 05:20:56 on dist-test-slave-42z9"
server_key: "e24070c15848290852328acb1c198f4e"
server_key_iv: "5c79d26745005fb3012dcc3c3b1614d2"
server_key_version: "encryptionkey@0"
I20260812 05:20:57.307646 13968 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786512017613366-32144-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786512017613366-32144-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786512017613366-32144-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 05:20:57.318642 13968 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 05:20:57.320344 13968 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 05:20:57.320472 13968 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 05:20:57.324235 13968 rpc_server.cc:307] RPC server started. Bound to: 127.31.100.62:41525
I20260812 05:20:57.324277 14102 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.100.62:41525 every 8 connection(s)
I20260812 05:20:57.324600 13968 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786512017613366-32144-0/minicluster-data/master-0/data/info.pb
I20260812 05:20:57.326010 14103 sys_catalog.cc:263] Verifying existing consensus state
I20260812 05:20:57.326614 14103 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a2275c27dce54c69bb2d43da953129a0: Bootstrap starting.
I20260812 05:20:57.327785 32144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0waFvT/build/release/bin/kudu as pid 13968
I20260812 05:20:57.328063 32144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0waFvT/build/release/bin/kudu
/tmp/dist-test-task0waFvT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786512017613366-32144-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786512017613366-32144-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786512017613366-32144-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786512017613366-32144-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.100.1:43473
--local_ip_for_outbound_sockets=127.31.100.1
--tserver_master_addrs=127.31.100.62:41525
--webserver_port=42253
--webserver_interface=127.31.100.1
--keytab_file=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.1.keytab
--principal=kudu/127.31.100.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.100.20:35097
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.31.100.1
--keytab_file=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786512017613366-32144-0/krb5kdc/oryx_127.31.100.1.keytab with env {KRB5CCNAME=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786512017613366-32144-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786512017613366-32144-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786512017613366-32144-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 05:20:57.335201 14103 log.cc:824] T 00000000000000000000000000000000 P a2275c27dce54c69bb2d43da953129a0: Log is configured to *not* fsync() on all Append() calls
I20260812 05:20:57.336773 14103 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a2275c27dce54c69bb2d43da953129a0: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 05:20:57.337033 14103 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a2275c27dce54c69bb2d43da953129a0: Bootstrap complete.
I20260812 05:20:57.339103 14103 raft_consensus.cc:348] T 00000000000000000000000000000000 P a2275c27dce54c69bb2d43da953129a0 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2275c27dce54c69bb2d43da953129a0" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 41525 } }
I20260812 05:20:57.339313 14103 raft_consensus.cc:729] T 00000000000000000000000000000000 P a2275c27dce54c69bb2d43da953129a0 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2275c27dce54c69bb2d43da953129a0, State: Initialized, Role: FOLLOWER
I20260812 05:20:57.339468 14103 consensus_queue.cc:260] T 00000000000000000000000000000000 P a2275c27dce54c69bb2d43da953129a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2275c27dce54c69bb2d43da953129a0" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 41525 } }
I20260812 05:20:57.339540 14103 raft_consensus.cc:388] T 00000000000000000000000000000000 P a2275c27dce54c69bb2d43da953129a0 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 05:20:57.339581 14103 raft_consensus.cc:482] T 00000000000000000000000000000000 P a2275c27dce54c69bb2d43da953129a0 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 05:20:57.339625 14103 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a2275c27dce54c69bb2d43da953129a0 [term 1 FOLLOWER]: Advancing to term 2
I20260812 05:20:57.340523 14103 raft_consensus.cc:504] T 00000000000000000000000000000000 P a2275c27dce54c69bb2d43da953129a0 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2275c27dce54c69bb2d43da953129a0" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 41525 } }
I20260812 05:20:57.340613 14103 leader_election.cc:302] T 00000000000000000000000000000000 P a2275c27dce54c69bb2d43da953129a0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a2275c27dce54c69bb2d43da953129a0; no voters:
I20260812 05:20:57.340757 14103 leader_election.cc:288] T 00000000000000000000000000000000 P a2275c27dce54c69bb2d43da953129a0 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260812 05:20:57.340809 14112 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a2275c27dce54c69bb2d43da953129a0 [term 2 FOLLOWER]: Leader election won for term 2
I20260812 05:20:57.340927 14112 raft_consensus.cc:686] T 00000000000000000000000000000000 P a2275c27dce54c69bb2d43da953129a0 [term 2 LEADER]: Becoming Leader. State: Replica: a2275c27dce54c69bb2d43da953129a0, State: Running, Role: LEADER
I20260812 05:20:57.341012 14112 consensus_queue.cc:237] T 00000000000000000000000000000000 P a2275c27dce54c69bb2d43da953129a0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2275c27dce54c69bb2d43da953129a0" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 41525 } }
I20260812 05:20:57.341019 14103 sys_catalog.cc:564] T 00000000000000000000000000000000 P a2275c27dce54c69bb2d43da953129a0 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 05:20:57.341840 14113 sys_catalog.cc:455] T 00000000000000000000000000000000 P a2275c27dce54c69bb2d43da953129a0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a2275c27dce54c69bb2d43da953129a0. Latest consensus state: current_term: 2 leader_uuid: "a2275c27dce54c69bb2d43da953129a0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2275c27dce54c69bb2d43da953129a0" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 41525 } } }
I20260812 05:20:57.341853 14114 sys_catalog.cc:455] T 00000000000000000000000000000000 P a2275c27dce54c69bb2d43da953129a0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "a2275c27dce54c69bb2d43da953129a0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2275c27dce54c69bb2d43da953129a0" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 41525 } } }
I20260812 05:20:57.341933 14114 sys_catalog.cc:458] T 00000000000000000000000000000000 P a2275c27dce54c69bb2d43da953129a0 [sys.catalog]: This master's current role is: LEADER
I20260812 05:20:57.341933 14113 sys_catalog.cc:458] T 00000000000000000000000000000000 P a2275c27dce54c69bb2d43da953129a0 [sys.catalog]: This master's current role is: LEADER
I20260812 05:20:57.342110 14123 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 05:20:57.342552 14123 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 05:20:57.342681 14123 catalog_manager.cc:1259] Loaded cluster ID: 75d8d0f990874adb8785d6c6d015c114
I20260812 05:20:57.342841 14123 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 05:20:57.343982 14123 catalog_manager.cc:1504] Loading token signing keys...
I20260812 05:20:57.344162 14123 catalog_manager.cc:5879] T 00000000000000000000000000000000 P a2275c27dce54c69bb2d43da953129a0: Loaded TSK: 0
I20260812 05:20:57.344369 14123 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260812 05:20:57.409404 14110 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 05:20:57.409637 14110 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 05:20:57.409663 14110 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 05:20:57.410877 14110 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 05:20:57.410918 14110 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 05:20:57.410943 14110 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.100.1
I20260812 05:20:57.412322 14110 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.100.20:35097
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786512017613366-32144-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786512017613366-32144-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786512017613366-32144-0/krb5kdc/oryx_127.31.100.1.keytab
--principal=oryx/127.31.100.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.100.1:43473
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786512017613366-32144-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.100.1
--webserver_port=42253
--tserver_master_addrs=127.31.100.62:41525
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14110
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.100.1
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786512017613366-32144-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 04:20:34 UTC on ccycloud.quasar-phqdqp.root.comops.site
build id 4581871
I20260812 05:20:57.412525 14110 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 05:20:57.412732 14110 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 05:20:57.418529 14140 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 05:20:57.418568 14139 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 05:20:57.418706 14110 server_base.cc:1034] running on GCE node
W20260812 05:20:57.418555 14145 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 05:20:57.418982 14110 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 05:20:57.419173 14110 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 05:20:57.420307 14110 hybrid_clock.cc:648] HybridClock initialized: now 1786512057420287 us; error 38 us; skew 500 ppm
Aug 12 05:20:57 dist-test-slave-42z9 krb5kdc[13098](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512057, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.100.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 05:20:57.422603 14110 init.cc:377] Logged in from keytab as oryx/127.31.100.1@KRBTEST.COM (short username oryx)
I20260812 05:20:57.423009 14110 webserver.cc:460] Webserver started at http://127.31.100.1:42253/ using document root <none> and password file <none>
I20260812 05:20:57.423228 14110 fs_manager.cc:359] Metadata directory not provided
I20260812 05:20:57.423291 14110 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 05:20:57.424589 14110 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260812 05:20:57.425168 14159 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 05:20:57.425347 14110 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260812 05:20:57.425467 14110 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786512017613366-32144-0/minicluster-data/ts-0/data,/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786512017613366-32144-0/minicluster-data/ts-0/wal
uuid: "3e68316b411745f69a83096255039bb1"
format_stamp: "Formatted at 2026-08-12 05:20:56 on dist-test-slave-42z9"
server_key: "31e3953ba354e405de805cb2e172153e"
server_key_iv: "a0c90013f506a5c6f2d59f37d88ce213"
server_key_version: "encryptionkey@0"
I20260812 05:20:57.425853 14110 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786512017613366-32144-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786512017613366-32144-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786512017613366-32144-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 05:20:57.441985 14110 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 05:20:57.444063 14110 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 05:20:57.444193 14110 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 05:20:57.444387 14110 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 05:20:57.444666 14110 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 05:20:57.444700 14110 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 05:20:57.444726 14110 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 05:20:57.444749 14110 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 05:20:57.450239 14110 rpc_server.cc:307] RPC server started. Bound to: 127.31.100.1:43473
I20260812 05:20:57.450313 14344 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.100.1:43473 every 8 connection(s)
I20260812 05:20:57.450599 14110 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786512017613366-32144-0/minicluster-data/ts-0/data/info.pb
I20260812 05:20:57.453663 32144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0waFvT/build/release/bin/kudu as pid 14110
I20260812 05:20:57.453969 32144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0waFvT/build/release/bin/kudu
/tmp/dist-test-task0waFvT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786512017613366-32144-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786512017613366-32144-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786512017613366-32144-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786512017613366-32144-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.100.2:35847
--local_ip_for_outbound_sockets=127.31.100.2
--tserver_master_addrs=127.31.100.62:41525
--webserver_port=38829
--webserver_interface=127.31.100.2
--keytab_file=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.2.keytab
--principal=kudu/127.31.100.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.100.20:35097
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.31.100.2
--keytab_file=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786512017613366-32144-0/krb5kdc/oryx_127.31.100.2.keytab with env {KRB5CCNAME=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786512017613366-32144-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786512017613366-32144-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786512017613366-32144-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 12 05:20:57 dist-test-slave-42z9 krb5kdc[13098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.100.1@KRBTEST.COM for oryx/127.31.100.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 05:20:57 dist-test-slave-42z9 krb5kdc[13098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.100.1@KRBTEST.COM for oryx/127.31.100.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 05:20:57.460119 14351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:57.451333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.62:41525 (local address 127.31.100.1:49805)
0812 05:20:57.451512 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0812 05:20:57.451551 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0812 05:20:57.452022 (+ 471us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:20:57.452410 (+ 388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:20:57.452414 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:20:57.452950 (+ 536us) client_negotiation.cc:189] Negotiated authn=SASL
0812 05:20:57.453125 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:20:57.453130 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:20:57.454590 (+ 1460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:20:57.454591 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:20:57.455771 (+ 1180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:20:57.455774 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:20:57.456650 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:20:57.456651 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:20:57.456774 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:57.457276 (+ 502us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 05:20:57.457285 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0812 05:20:57.459925 (+ 2640us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.100.62:41525: Server oryx/127.31.100.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":94,"threads_started":1}
W20260812 05:20:57.460314 14345 heartbeater.cc:644] Failed to heartbeat to 127.31.100.62:41525 (0 consecutive failures): Not authorized: Failed to ping master at 127.31.100.62:41525: Client connection negotiation failed: client connection to 127.31.100.62:41525: Server oryx/127.31.100.62@KRBTEST.COM not found in Kerberos database
W20260812 05:20:57.460704 14353 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 05:20:57.451425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.100.1:49805 (local address 127.31.100.62:41525)
0812 05:20:57.451619 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0812 05:20:57.451628 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 05:20:57.451666 (+ 38us) server_negotiation.cc:407] Connection header received
0812 05:20:57.452209 (+ 543us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:20:57.452213 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:20:57.452254 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:20:57.452362 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:20:57.453242 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:57.454480 (+ 1238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:57.455873 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:57.456467 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:57.456566 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:57.460092 (+ 3526us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.100.1:49805: BlockingRecv error: recv got EOF from 127.31.100.1:49805 (error 108)
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":100,"threads_started":1}
Aug 12 05:20:57 dist-test-slave-42z9 krb5kdc[13098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.100.1@KRBTEST.COM for oryx/127.31.100.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 05:20:57 dist-test-slave-42z9 krb5kdc[13098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.100.1@KRBTEST.COM for oryx/127.31.100.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 05:20:57.466410 14351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:57.460507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.62:41525 (local address 127.31.100.1:53041)
0812 05:20:57.461512 (+ 1005us) negotiation.cc:107] Waiting for socket to connect
0812 05:20:57.461524 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0812 05:20:57.461579 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:20:57.461732 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:20:57.461733 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:20:57.461867 (+ 134us) client_negotiation.cc:189] Negotiated authn=SASL
0812 05:20:57.461918 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:20:57.461920 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:20:57.462791 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:20:57.462792 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:20:57.463547 (+ 755us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:20:57.463550 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:20:57.464015 (+ 465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:20:57.464016 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:20:57.464085 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:57.464477 (+ 392us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 05:20:57.464483 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0812 05:20:57.466348 (+ 1865us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.100.62:41525: Server oryx/127.31.100.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":986}
W20260812 05:20:57.466629 14355 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 05:20:57.460558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.100.1:53041 (local address 127.31.100.62:41525)
0812 05:20:57.460710 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0812 05:20:57.460712 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 05:20:57.461577 (+ 865us) server_negotiation.cc:407] Connection header received
0812 05:20:57.461631 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:20:57.461632 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:20:57.461644 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:20:57.461723 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:20:57.461988 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:57.462710 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:57.463634 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:57.463932 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:57.464006 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:57.466437 (+ 2431us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.100.1:53041: BlockingRecv error: recv got EOF from 127.31.100.1:53041 (error 108)
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":86,"threads_started":1}
Aug 12 05:20:57 dist-test-slave-42z9 krb5kdc[13098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.100.1@KRBTEST.COM for oryx/127.31.100.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 05:20:57 dist-test-slave-42z9 krb5kdc[13098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.100.1@KRBTEST.COM for oryx/127.31.100.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 05:20:57.473762 14351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:57.466622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.62:41525 (local address 127.31.100.1:50199)
0812 05:20:57.469496 (+ 2874us) negotiation.cc:107] Waiting for socket to connect
0812 05:20:57.469506 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0812 05:20:57.469558 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:20:57.469679 (+ 121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:20:57.469679 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:20:57.469813 (+ 134us) client_negotiation.cc:189] Negotiated authn=SASL
0812 05:20:57.469845 (+ 32us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:20:57.469846 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:20:57.470627 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:20:57.470628 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:20:57.471410 (+ 782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:20:57.471412 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:20:57.471723 (+ 311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:20:57.471724 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:20:57.471774 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:57.472094 (+ 320us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 05:20:57.472099 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0812 05:20:57.473700 (+ 1601us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.100.62:41525: Server oryx/127.31.100.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":2857}
W20260812 05:20:57.473817 14353 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 05:20:57.466676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.100.1:50199 (local address 127.31.100.62:41525)
0812 05:20:57.466708 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0812 05:20:57.466709 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0812 05:20:57.469558 (+ 2849us) server_negotiation.cc:407] Connection header received
0812 05:20:57.469603 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:20:57.469604 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:20:57.469616 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:20:57.469670 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:20:57.469902 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:57.470551 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:57.471488 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:57.471664 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:57.471716 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:57.473771 (+ 2055us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.100.1:50199: BlockingRecv error: recv got EOF from 127.31.100.1:50199 (error 108)
Metrics: {"server-negotiator.queue_time_us":15}
W20260812 05:20:57.473871 14345 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260812 05:20:57.539327 14354 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 05:20:57.539584 14354 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 05:20:57.539608 14354 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 05:20:57.540911 14354 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 05:20:57.540993 14354 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 05:20:57.541021 14354 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.100.2
I20260812 05:20:57.542636 14354 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.100.20:35097
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786512017613366-32144-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786512017613366-32144-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786512017613366-32144-0/krb5kdc/oryx_127.31.100.2.keytab
--principal=oryx/127.31.100.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.100.2:35847
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786512017613366-32144-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.100.2
--webserver_port=38829
--tserver_master_addrs=127.31.100.62:41525
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14354
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.100.2
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786512017613366-32144-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 04:20:34 UTC on ccycloud.quasar-phqdqp.root.comops.site
build id 4581871
I20260812 05:20:57.542860 14354 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 05:20:57.543080 14354 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 05:20:57.548154 14366 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 05:20:57.548130 14363 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 05:20:57.548249 14354 server_base.cc:1034] running on GCE node
W20260812 05:20:57.548174 14362 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 05:20:57.548488 14354 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 05:20:57.548707 14354 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 05:20:57.549897 14354 hybrid_clock.cc:648] HybridClock initialized: now 1786512057549876 us; error 34 us; skew 500 ppm
Aug 12 05:20:57 dist-test-slave-42z9 krb5kdc[13098](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512057, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.100.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 05:20:57.552238 14354 init.cc:377] Logged in from keytab as oryx/127.31.100.2@KRBTEST.COM (short username oryx)
I20260812 05:20:57.552644 14354 webserver.cc:460] Webserver started at http://127.31.100.2:38829/ using document root <none> and password file <none>
I20260812 05:20:57.552838 14354 fs_manager.cc:359] Metadata directory not provided
I20260812 05:20:57.552894 14354 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 05:20:57.554234 14354 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260812 05:20:57.554749 14377 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 05:20:57.554891 14354 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260812 05:20:57.554986 14354 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786512017613366-32144-0/minicluster-data/ts-1/data,/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786512017613366-32144-0/minicluster-data/ts-1/wal
uuid: "7e63bace79bd4e3e81b406d0efe31082"
format_stamp: "Formatted at 2026-08-12 05:20:56 on dist-test-slave-42z9"
server_key: "ca8758c68d1cf5f0c443ef856117722d"
server_key_iv: "de15eb999882a609b66e1fec3020dc60"
server_key_version: "encryptionkey@0"
I20260812 05:20:57.555300 14354 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786512017613366-32144-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786512017613366-32144-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786512017613366-32144-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 05:20:57.577917 14354 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 05:20:57.579994 14354 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 05:20:57.580132 14354 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 05:20:57.580349 14354 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 05:20:57.580638 14354 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 05:20:57.580672 14354 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 05:20:57.580694 14354 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 05:20:57.580717 14354 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 05:20:57.586301 14354 rpc_server.cc:307] RPC server started. Bound to: 127.31.100.2:35847
I20260812 05:20:57.586356 14564 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.100.2:35847 every 8 connection(s)
I20260812 05:20:57.586674 14354 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786512017613366-32144-0/minicluster-data/ts-1/data/info.pb
I20260812 05:20:57.589381 32144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0waFvT/build/release/bin/kudu as pid 14354
I20260812 05:20:57.589720 32144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0waFvT/build/release/bin/kudu
/tmp/dist-test-task0waFvT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786512017613366-32144-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786512017613366-32144-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786512017613366-32144-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786512017613366-32144-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.100.3:36375
--local_ip_for_outbound_sockets=127.31.100.3
--tserver_master_addrs=127.31.100.62:41525
--webserver_port=38369
--webserver_interface=127.31.100.3
--keytab_file=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.3.keytab
--principal=kudu/127.31.100.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.100.20:35097
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.31.100.3
--keytab_file=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786512017613366-32144-0/krb5kdc/oryx_127.31.100.3.keytab with env {KRB5CCNAME=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786512017613366-32144-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786512017613366-32144-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786512017613366-32144-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 12 05:20:57 dist-test-slave-42z9 krb5kdc[13098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.100.2@KRBTEST.COM for oryx/127.31.100.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 05:20:57 dist-test-slave-42z9 krb5kdc[13098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.100.2@KRBTEST.COM for oryx/127.31.100.62@KRBTEST.COM, Server not found in Kerberos database
W20260812 05:20:57.594187 14353 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 05:20:57.587304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.100.2:45051 (local address 127.31.100.62:41525)
0812 05:20:57.587387 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0812 05:20:57.587390 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 05:20:57.587569 (+ 179us) server_negotiation.cc:407] Connection header received
0812 05:20:57.588038 (+ 469us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:20:57.588039 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:20:57.588053 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:20:57.588097 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:20:57.588960 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:57.589459 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:57.590792 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:57.590998 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:57.591060 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:57.594130 (+ 3070us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.100.2:45051: BlockingRecv error: recv got EOF from 127.31.100.2:45051 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
I20260812 05:20:57.594214 14567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:57.587293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.62:41525 (local address 127.31.100.2:45051)
0812 05:20:57.587450 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0812 05:20:57.587486 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0812 05:20:57.587962 (+ 476us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:20:57.588157 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:20:57.588159 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:20:57.588702 (+ 543us) client_negotiation.cc:189] Negotiated authn=SASL
0812 05:20:57.588887 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:20:57.588892 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:20:57.589546 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:20:57.589547 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:20:57.590672 (+ 1125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:20:57.590676 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:20:57.591083 (+ 407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:20:57.591084 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:20:57.591223 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:57.591729 (+ 506us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 05:20:57.591737 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0812 05:20:57.594023 (+ 2286us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.100.62:41525: Server oryx/127.31.100.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":89,"threads_started":1}
W20260812 05:20:57.594417 14565 heartbeater.cc:644] Failed to heartbeat to 127.31.100.62:41525 (0 consecutive failures): Not authorized: Failed to ping master at 127.31.100.62:41525: Client connection negotiation failed: client connection to 127.31.100.62:41525: Server oryx/127.31.100.62@KRBTEST.COM not found in Kerberos database
Aug 12 05:20:57 dist-test-slave-42z9 krb5kdc[13098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.100.2@KRBTEST.COM for oryx/127.31.100.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 05:20:57 dist-test-slave-42z9 krb5kdc[13098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.100.2@KRBTEST.COM for oryx/127.31.100.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 05:20:57.598539 14567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:57.594652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.62:41525 (local address 127.31.100.2:41509)
0812 05:20:57.594693 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0812 05:20:57.594702 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0812 05:20:57.594736 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:20:57.594852 (+ 116us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:20:57.594853 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:20:57.594971 (+ 118us) client_negotiation.cc:189] Negotiated authn=SASL
0812 05:20:57.595012 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:20:57.595013 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:20:57.595634 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:20:57.595635 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:20:57.596119 (+ 484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:20:57.596121 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:20:57.596433 (+ 312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:20:57.596434 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:20:57.596494 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:57.596892 (+ 398us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 05:20:57.596896 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0812 05:20:57.598479 (+ 1583us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.100.62:41525: Server oryx/127.31.100.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":25}
W20260812 05:20:57.598619 14353 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 05:20:57.594709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.100.2:41509 (local address 127.31.100.62:41525)
0812 05:20:57.594762 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0812 05:20:57.594764 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 05:20:57.594772 (+ 8us) server_negotiation.cc:407] Connection header received
0812 05:20:57.594793 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:20:57.594794 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:20:57.594805 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:20:57.594850 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:20:57.595076 (+ 226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:57.595560 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:57.596188 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:57.596376 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:57.596428 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:57.598566 (+ 2138us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.100.2:41509: BlockingRecv error: recv got EOF from 127.31.100.2:41509 (error 108)
Metrics: {"server-negotiator.queue_time_us":31}
Aug 12 05:20:57 dist-test-slave-42z9 krb5kdc[13098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.100.2@KRBTEST.COM for oryx/127.31.100.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 05:20:57 dist-test-slave-42z9 krb5kdc[13098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.100.2@KRBTEST.COM for oryx/127.31.100.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 05:20:57.602541 14567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:57.598765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.62:41525 (local address 127.31.100.2:34239)
0812 05:20:57.598886 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0812 05:20:57.598895 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0812 05:20:57.598938 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:20:57.599049 (+ 111us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:20:57.599049 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:20:57.599164 (+ 115us) client_negotiation.cc:189] Negotiated authn=SASL
0812 05:20:57.599188 (+ 24us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:20:57.599189 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:20:57.599769 (+ 580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:20:57.599770 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:20:57.600235 (+ 465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:20:57.600236 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:20:57.600543 (+ 307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:20:57.600544 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:20:57.600592 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:57.600995 (+ 403us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 05:20:57.600998 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0812 05:20:57.602481 (+ 1483us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.100.62:41525: Server oryx/127.31.100.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":82,"threads_started":1}
W20260812 05:20:57.602612 14353 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 05:20:57.598822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.100.2:34239 (local address 127.31.100.62:41525)
0812 05:20:57.598858 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0812 05:20:57.598859 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0812 05:20:57.598928 (+ 69us) server_negotiation.cc:407] Connection header received
0812 05:20:57.598994 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:20:57.598995 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:20:57.599003 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:20:57.599056 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:20:57.599248 (+ 192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:57.599701 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:57.600300 (+ 599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:57.600486 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:57.600538 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:57.602563 (+ 2025us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.100.2:34239: BlockingRecv error: recv got EOF from 127.31.100.2:34239 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260812 05:20:57.602666 14565 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260812 05:20:57.672071 14568 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 05:20:57.672256 14568 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 05:20:57.672276 14568 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 05:20:57.673399 14568 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 05:20:57.673468 14568 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 05:20:57.673511 14568 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.100.3
I20260812 05:20:57.674926 14568 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.100.20:35097
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786512017613366-32144-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786512017613366-32144-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786512017613366-32144-0/krb5kdc/oryx_127.31.100.3.keytab
--principal=oryx/127.31.100.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.100.3:36375
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786512017613366-32144-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.100.3
--webserver_port=38369
--tserver_master_addrs=127.31.100.62:41525
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14568
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.100.3
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786512017613366-32144-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 04:20:34 UTC on ccycloud.quasar-phqdqp.root.comops.site
build id 4581871
I20260812 05:20:57.675158 14568 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 05:20:57.675364 14568 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 05:20:57.680159 14583 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 05:20:57.680181 14578 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 05:20:57.680183 14581 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 05:20:57.680279 14568 server_base.cc:1034] running on GCE node
I20260812 05:20:57.680563 14568 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 05:20:57.680786 14568 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 05:20:57.681921 14568 hybrid_clock.cc:648] HybridClock initialized: now 1786512057681909 us; error 28 us; skew 500 ppm
Aug 12 05:20:57 dist-test-slave-42z9 krb5kdc[13098](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512057, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.100.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 05:20:57.684408 14568 init.cc:377] Logged in from keytab as oryx/127.31.100.3@KRBTEST.COM (short username oryx)
I20260812 05:20:57.684800 14568 webserver.cc:460] Webserver started at http://127.31.100.3:38369/ using document root <none> and password file <none>
I20260812 05:20:57.685020 14568 fs_manager.cc:359] Metadata directory not provided
I20260812 05:20:57.685091 14568 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 05:20:57.686460 14568 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260812 05:20:57.687059 14591 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 05:20:57.687227 14568 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260812 05:20:57.687316 14568 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786512017613366-32144-0/minicluster-data/ts-2/data,/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786512017613366-32144-0/minicluster-data/ts-2/wal
uuid: "7e98f49f4a8345b9a9b24bbf55623d3e"
format_stamp: "Formatted at 2026-08-12 05:20:57 on dist-test-slave-42z9"
server_key: "56c4adb935a0a71162e0f2901359ccf2"
server_key_iv: "8362a45092d8684e1e6670fafb65ad20"
server_key_version: "encryptionkey@0"
I20260812 05:20:57.687637 14568 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786512017613366-32144-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786512017613366-32144-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786512017613366-32144-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 05:20:57.706802 14568 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 05:20:57.708491 14568 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 05:20:57.708616 14568 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 05:20:57.708811 14568 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 05:20:57.709089 14568 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 05:20:57.709122 14568 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 05:20:57.709149 14568 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 05:20:57.709178 14568 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 05:20:57.714754 14568 rpc_server.cc:307] RPC server started. Bound to: 127.31.100.3:36375
I20260812 05:20:57.714800 14775 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.100.3:36375 every 8 connection(s)
I20260812 05:20:57.715122 14568 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786512017613366-32144-0/minicluster-data/ts-2/data/info.pb
I20260812 05:20:57.715700 32144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0waFvT/build/release/bin/kudu as pid 14568
Aug 12 05:20:57 dist-test-slave-42z9 krb5kdc[13098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.100.3@KRBTEST.COM for oryx/127.31.100.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 05:20:57 dist-test-slave-42z9 krb5kdc[13098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.100.3@KRBTEST.COM for oryx/127.31.100.62@KRBTEST.COM, Server not found in Kerberos database
W20260812 05:20:57.721913 14353 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 05:20:57.715789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.100.3:33905 (local address 127.31.100.62:41525)
0812 05:20:57.716014 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0812 05:20:57.716017 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 05:20:57.716066 (+ 49us) server_negotiation.cc:407] Connection header received
0812 05:20:57.716578 (+ 512us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:20:57.716579 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:20:57.716590 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:20:57.716640 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:20:57.717510 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:57.717975 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:57.718910 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:57.719093 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:57.719145 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:57.721858 (+ 2713us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.100.3:33905: BlockingRecv error: recv got EOF from 127.31.100.3:33905 (error 108)
Metrics: {"server-negotiator.queue_time_us":196}
I20260812 05:20:57.721966 14782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:57.715795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.62:41525 (local address 127.31.100.3:33905)
0812 05:20:57.715961 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0812 05:20:57.715998 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0812 05:20:57.716484 (+ 486us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:20:57.716706 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:20:57.716709 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:20:57.717232 (+ 523us) client_negotiation.cc:189] Negotiated authn=SASL
0812 05:20:57.717408 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:20:57.717413 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:20:57.718053 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:20:57.718055 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:20:57.718833 (+ 778us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:20:57.718835 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:20:57.719160 (+ 325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:20:57.719161 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:20:57.719270 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:57.719719 (+ 449us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 05:20:57.719727 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0812 05:20:57.721779 (+ 2052us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.100.62:41525: Server oryx/127.31.100.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":78,"threads_started":1}
W20260812 05:20:57.722347 14777 heartbeater.cc:644] Failed to heartbeat to 127.31.100.62:41525 (0 consecutive failures): Not authorized: Failed to ping master at 127.31.100.62:41525: Client connection negotiation failed: client connection to 127.31.100.62:41525: Server oryx/127.31.100.62@KRBTEST.COM not found in Kerberos database
I20260812 05:20:57.723084 14355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:57.716409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56658 (local address 127.31.100.62:41525)
0812 05:20:57.716455 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0812 05:20:57.716457 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 05:20:57.716463 (+ 6us) server_negotiation.cc:407] Connection header received
0812 05:20:57.716485 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:20:57.716486 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:20:57.716495 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:20:57.716540 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:20:57.717115 (+ 575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:57.717679 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:57.718422 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:57.718618 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:57.718673 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:57.719929 (+ 1256us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 05:20:57.719933 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 05:20:57.719934 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 05:20:57.719945 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0812 05:20:57.721592 (+ 1647us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:20:57.721917 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:20:57.721921 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:20:57.721922 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:20:57.722041 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:20:57.722238 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:20:57.722238 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:20:57.722239 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:20:57.722508 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 05:20:57.722646 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0812 05:20:57.722890 (+ 244us) server_negotiation.cc:299] Negotiation successful
0812 05:20:57.722998 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
Aug 12 05:20:57 dist-test-slave-42z9 krb5kdc[13098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.100.3@KRBTEST.COM for oryx/127.31.100.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 05:20:57 dist-test-slave-42z9 krb5kdc[13098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.100.3@KRBTEST.COM for oryx/127.31.100.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 05:20:57.726402 14782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:57.722572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.62:41525 (local address 127.31.100.3:51743)
0812 05:20:57.722612 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0812 05:20:57.722622 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0812 05:20:57.722658 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:20:57.722769 (+ 111us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:20:57.722770 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:20:57.722912 (+ 142us) client_negotiation.cc:189] Negotiated authn=SASL
0812 05:20:57.722977 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:20:57.722979 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:20:57.723688 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:20:57.723689 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:20:57.724250 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:20:57.724252 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:20:57.724544 (+ 292us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:20:57.724544 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:20:57.724587 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:57.724869 (+ 282us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 05:20:57.724872 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0812 05:20:57.726361 (+ 1489us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.100.62:41525: Server oryx/127.31.100.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":24}
W20260812 05:20:57.726502 14353 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 05:20:57.722617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.100.3:51743 (local address 127.31.100.62:41525)
0812 05:20:57.722662 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0812 05:20:57.722663 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0812 05:20:57.722685 (+ 22us) server_negotiation.cc:407] Connection header received
0812 05:20:57.722703 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:20:57.722704 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:20:57.722712 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:20:57.722754 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:20:57.723059 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:57.723626 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:57.724314 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:57.724493 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:57.724541 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:57.726442 (+ 1901us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.100.3:51743: BlockingRecv error: recv got EOF from 127.31.100.3:51743 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
Aug 12 05:20:57 dist-test-slave-42z9 krb5kdc[13098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.100.3@KRBTEST.COM for oryx/127.31.100.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 05:20:57 dist-test-slave-42z9 krb5kdc[13098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.100.3@KRBTEST.COM for oryx/127.31.100.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 05:20:57.730170 14782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:57.726638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.62:41525 (local address 127.31.100.3:38331)
0812 05:20:57.726723 (+ 85us) negotiation.cc:107] Waiting for socket to connect
0812 05:20:57.726734 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0812 05:20:57.726775 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:20:57.726873 (+ 98us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:20:57.726874 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:20:57.726984 (+ 110us) client_negotiation.cc:189] Negotiated authn=SASL
0812 05:20:57.727009 (+ 25us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:20:57.727010 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:20:57.727573 (+ 563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:20:57.727575 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:20:57.728051 (+ 476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:20:57.728053 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:20:57.728339 (+ 286us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:20:57.728340 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:20:57.728375 (+ 35us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:57.728709 (+ 334us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 05:20:57.728714 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0812 05:20:57.730127 (+ 1413us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.100.62:41525: Server oryx/127.31.100.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":67}
W20260812 05:20:57.730262 14353 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 05:20:57.726682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.100.3:38331 (local address 127.31.100.62:41525)
0812 05:20:57.726720 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0812 05:20:57.726721 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0812 05:20:57.726763 (+ 42us) server_negotiation.cc:407] Connection header received
0812 05:20:57.726810 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:20:57.726810 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:20:57.726817 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:20:57.726854 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:20:57.727055 (+ 201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:57.727491 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:57.728122 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:57.728289 (+ 167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:57.728334 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:57.730206 (+ 1872us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.100.3:38331: BlockingRecv error: recv got EOF from 127.31.100.3:38331 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260812 05:20:57.730273 14777 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Aug 12 05:20:58 dist-test-slave-42z9 krb5kdc[13098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.100.1@KRBTEST.COM for oryx/127.31.100.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 05:20:58 dist-test-slave-42z9 krb5kdc[13098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.100.1@KRBTEST.COM for oryx/127.31.100.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 05:20:58.478550 14788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:58.474222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.62:41525 (local address 127.31.100.1:32899)
0812 05:20:58.474344 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0812 05:20:58.474356 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0812 05:20:58.474403 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:20:58.474544 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:20:58.474544 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:20:58.474699 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0812 05:20:58.474751 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:20:58.474753 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:20:58.475415 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:20:58.475416 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:20:58.475970 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:20:58.475972 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:20:58.476288 (+ 316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:20:58.476289 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:20:58.476331 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:58.476774 (+ 443us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 05:20:58.476778 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0812 05:20:58.478486 (+ 1708us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.100.62:41525: Server oryx/127.31.100.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":64,"threads_started":1}
W20260812 05:20:58.478689 14789 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 05:20:58.474298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.100.1:32899 (local address 127.31.100.62:41525)
0812 05:20:58.474431 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0812 05:20:58.474432 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0812 05:20:58.474440 (+ 8us) server_negotiation.cc:407] Connection header received
0812 05:20:58.474460 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:20:58.474461 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:20:58.474474 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:20:58.474535 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:20:58.474833 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:58.475344 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:58.476038 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:58.476232 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:58.476294 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:58.478603 (+ 2309us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.100.1:32899: BlockingRecv error: recv got EOF from 127.31.100.1:32899 (error 108)
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":65,"threads_started":1}
Aug 12 05:20:58 dist-test-slave-42z9 krb5kdc[13098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.100.2@KRBTEST.COM for oryx/127.31.100.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 05:20:58 dist-test-slave-42z9 krb5kdc[13098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.100.2@KRBTEST.COM for oryx/127.31.100.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 05:20:58.607563 14790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:58.603049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.62:41525 (local address 127.31.100.2:53841)
0812 05:20:58.603166 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0812 05:20:58.603178 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0812 05:20:58.603238 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:20:58.603355 (+ 117us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:20:58.603356 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:20:58.603523 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0812 05:20:58.603573 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:20:58.603575 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:20:58.604240 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:20:58.604241 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:20:58.604825 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:20:58.604827 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:20:58.605146 (+ 319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:20:58.605147 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:20:58.605194 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:58.605691 (+ 497us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 05:20:58.605697 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0812 05:20:58.607474 (+ 1777us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.100.62:41525: Server oryx/127.31.100.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":58,"threads_started":1}
W20260812 05:20:58.607625 14789 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 05:20:58.603125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.100.2:53841 (local address 127.31.100.62:41525)
0812 05:20:58.603212 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0812 05:20:58.603214 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 05:20:58.603223 (+ 9us) server_negotiation.cc:407] Connection header received
0812 05:20:58.603290 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:20:58.603291 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:20:58.603301 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:20:58.603344 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:20:58.603660 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:58.604160 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:58.604891 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:58.605076 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:58.605123 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:58.607574 (+ 2451us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.100.2:53841: BlockingRecv error: recv got EOF from 127.31.100.2:53841 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Aug 12 05:20:58 dist-test-slave-42z9 krb5kdc[13098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.100.3@KRBTEST.COM for oryx/127.31.100.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 05:20:58 dist-test-slave-42z9 krb5kdc[13098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.100.3@KRBTEST.COM for oryx/127.31.100.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 05:20:58.734990 14791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:58.730576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.62:41525 (local address 127.31.100.3:59617)
0812 05:20:58.730689 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0812 05:20:58.730700 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0812 05:20:58.730763 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:20:58.730874 (+ 111us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:20:58.730874 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:20:58.731032 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0812 05:20:58.731083 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:20:58.731085 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:20:58.731712 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:20:58.731712 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:20:58.732250 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:20:58.732251 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:20:58.732631 (+ 380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:20:58.732632 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:20:58.732668 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:58.733101 (+ 433us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 05:20:58.733106 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0812 05:20:58.734897 (+ 1791us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.100.62:41525: Server oryx/127.31.100.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":65,"threads_started":1}
W20260812 05:20:58.735046 14789 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 05:20:58.730636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.100.3:59617 (local address 127.31.100.62:41525)
0812 05:20:58.730694 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0812 05:20:58.730696 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 05:20:58.730760 (+ 64us) server_negotiation.cc:407] Connection header received
0812 05:20:58.730816 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:20:58.730817 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:20:58.730828 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:20:58.730862 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:20:58.731137 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:58.731651 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:58.732342 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:58.732564 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:58.732633 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:58.734992 (+ 2359us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.100.3:59617: BlockingRecv error: recv got EOF from 127.31.100.3:59617 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
Aug 12 05:20:59 dist-test-slave-42z9 krb5kdc[13098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.100.1@KRBTEST.COM for oryx/127.31.100.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 05:20:59 dist-test-slave-42z9 krb5kdc[13098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.100.1@KRBTEST.COM for oryx/127.31.100.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 05:20:59.483511 14796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:59.478971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.62:41525 (local address 127.31.100.1:38659)
0812 05:20:59.479103 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0812 05:20:59.479114 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0812 05:20:59.479168 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:20:59.479296 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:20:59.479296 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:20:59.479469 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0812 05:20:59.479521 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:20:59.479523 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:20:59.480123 (+ 600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:20:59.480124 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:20:59.480713 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:20:59.480715 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:20:59.481081 (+ 366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:20:59.481082 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:20:59.481122 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:59.481656 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 05:20:59.481662 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0812 05:20:59.483423 (+ 1761us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.100.62:41525: Server oryx/127.31.100.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":71,"threads_started":1}
W20260812 05:20:59.483584 14797 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 05:20:59.479045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.100.1:38659 (local address 127.31.100.62:41525)
0812 05:20:59.479158 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0812 05:20:59.479160 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 05:20:59.479169 (+ 9us) server_negotiation.cc:407] Connection header received
0812 05:20:59.479226 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:20:59.479227 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:20:59.479238 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:20:59.479285 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:20:59.479575 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:59.480068 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:59.480807 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:59.481020 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:59.481076 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:59.483523 (+ 2447us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.100.1:38659: BlockingRecv error: recv got EOF from 127.31.100.1:38659 (error 108)
Metrics: {"server-negotiator.queue_time_us":91,"thread_start_us":51,"threads_started":1}
Aug 12 05:20:59 dist-test-slave-42z9 krb5kdc[13098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.100.2@KRBTEST.COM for oryx/127.31.100.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 05:20:59 dist-test-slave-42z9 krb5kdc[13098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.100.2@KRBTEST.COM for oryx/127.31.100.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 05:20:59.612704 14798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:59.608091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.62:41525 (local address 127.31.100.2:46751)
0812 05:20:59.608219 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0812 05:20:59.608229 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0812 05:20:59.608284 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:20:59.608403 (+ 119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:20:59.608404 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:20:59.608566 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0812 05:20:59.608617 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:20:59.608619 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:20:59.609239 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:20:59.609240 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:20:59.609857 (+ 617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:20:59.609860 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:20:59.610197 (+ 337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:20:59.610198 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:20:59.610259 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:59.610758 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 05:20:59.610763 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0812 05:20:59.612626 (+ 1863us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.100.62:41525: Server oryx/127.31.100.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":71,"threads_started":1}
W20260812 05:20:59.612785 14797 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 05:20:59.608217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.100.2:46751 (local address 127.31.100.62:41525)
0812 05:20:59.608278 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0812 05:20:59.608280 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 05:20:59.608290 (+ 10us) server_negotiation.cc:407] Connection header received
0812 05:20:59.608330 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:20:59.608330 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:20:59.608342 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:20:59.608387 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:20:59.608670 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:59.609170 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:59.609928 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:59.610128 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:59.610185 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:59.612732 (+ 2547us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.100.2:46751: BlockingRecv error: recv got EOF from 127.31.100.2:46751 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
Aug 12 05:20:59 dist-test-slave-42z9 krb5kdc[13098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.100.3@KRBTEST.COM for oryx/127.31.100.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 05:20:59 dist-test-slave-42z9 krb5kdc[13098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.100.3@KRBTEST.COM for oryx/127.31.100.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 05:20:59.740097 14799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:20:59.735516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.62:41525 (local address 127.31.100.3:50703)
0812 05:20:59.735672 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0812 05:20:59.735685 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 05:20:59.735741 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:20:59.735847 (+ 106us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:20:59.735847 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:20:59.736023 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0812 05:20:59.736074 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:20:59.736075 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:20:59.736700 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:20:59.736701 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:20:59.737242 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:20:59.737244 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:20:59.737622 (+ 378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:20:59.737623 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:20:59.737688 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:59.738126 (+ 438us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 05:20:59.738131 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0812 05:20:59.740040 (+ 1909us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.100.62:41525: Server oryx/127.31.100.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":77,"threads_started":1}
W20260812 05:20:59.740165 14797 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 05:20:59.735665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.100.3:50703 (local address 127.31.100.62:41525)
0812 05:20:59.735722 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0812 05:20:59.735724 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 05:20:59.735732 (+ 8us) server_negotiation.cc:407] Connection header received
0812 05:20:59.735792 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:20:59.735793 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:20:59.735804 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:20:59.735840 (+ 36us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:20:59.736122 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:59.736613 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:59.737316 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:20:59.737553 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:20:59.737622 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:20:59.740113 (+ 2491us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.100.3:50703: BlockingRecv error: recv got EOF from 127.31.100.3:50703 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
Aug 12 05:21:00 dist-test-slave-42z9 krb5kdc[13098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.100.1@KRBTEST.COM for oryx/127.31.100.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 05:21:00 dist-test-slave-42z9 krb5kdc[13098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.100.1@KRBTEST.COM for oryx/127.31.100.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 05:21:00.488440 14801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:21:00.484007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.62:41525 (local address 127.31.100.1:49953)
0812 05:21:00.484141 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0812 05:21:00.484152 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0812 05:21:00.484207 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:21:00.484330 (+ 123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:21:00.484331 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:21:00.484498 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0812 05:21:00.484550 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:21:00.484552 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:21:00.485185 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:21:00.485186 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:21:00.485766 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:21:00.485768 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:21:00.486135 (+ 367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:21:00.486136 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:21:00.486198 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:21:00.486606 (+ 408us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 05:21:00.486610 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0812 05:21:00.488334 (+ 1724us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.100.62:41525: Server oryx/127.31.100.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":74,"threads_started":1}
W20260812 05:21:00.488494 14802 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 05:21:00.484088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.100.1:49953 (local address 127.31.100.62:41525)
0812 05:21:00.484203 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0812 05:21:00.484205 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 05:21:00.484215 (+ 10us) server_negotiation.cc:407] Connection header received
0812 05:21:00.484253 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:21:00.484254 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:21:00.484265 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:21:00.484319 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:21:00.484601 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:21:00.485102 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:21:00.485827 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:21:00.486058 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:21:00.486115 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:21:00.488432 (+ 2317us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.100.1:49953: BlockingRecv error: recv got EOF from 127.31.100.1:49953 (error 108)
Metrics: {"server-negotiator.queue_time_us":93,"thread_start_us":59,"threads_started":1}
Aug 12 05:21:00 dist-test-slave-42z9 krb5kdc[13098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.100.2@KRBTEST.COM for oryx/127.31.100.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 05:21:00 dist-test-slave-42z9 krb5kdc[13098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.100.2@KRBTEST.COM for oryx/127.31.100.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 05:21:00.617353 14804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:21:00.613150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.62:41525 (local address 127.31.100.2:42777)
0812 05:21:00.613274 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0812 05:21:00.613285 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0812 05:21:00.613339 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:21:00.613470 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:21:00.613471 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:21:00.613643 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0812 05:21:00.613693 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:21:00.613694 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:21:00.614298 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:21:00.614299 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:21:00.614823 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:21:00.614824 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:21:00.615124 (+ 300us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:21:00.615125 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:21:00.615162 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:21:00.615579 (+ 417us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 05:21:00.615583 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0812 05:21:00.617294 (+ 1711us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.100.62:41525: Server oryx/127.31.100.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":71,"threads_started":1}
W20260812 05:21:00.617462 14802 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 05:21:00.613228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.100.2:42777 (local address 127.31.100.62:41525)
0812 05:21:00.613294 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0812 05:21:00.613296 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 05:21:00.613327 (+ 31us) server_negotiation.cc:407] Connection header received
0812 05:21:00.613385 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:21:00.613386 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:21:00.613396 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:21:00.613462 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:21:00.613746 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:21:00.614238 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:21:00.614887 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:21:00.615074 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:21:00.615120 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:21:00.617385 (+ 2265us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.100.2:42777: BlockingRecv error: recv got EOF from 127.31.100.2:42777 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Aug 12 05:21:00 dist-test-slave-42z9 krb5kdc[13098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.100.3@KRBTEST.COM for oryx/127.31.100.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 05:21:00 dist-test-slave-42z9 krb5kdc[13098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.100.3@KRBTEST.COM for oryx/127.31.100.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 05:21:00.744797 14806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:21:00.740549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.62:41525 (local address 127.31.100.3:46057)
0812 05:21:00.740670 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0812 05:21:00.740681 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0812 05:21:00.740739 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:21:00.740851 (+ 112us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:21:00.740852 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:21:00.741018 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0812 05:21:00.741072 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:21:00.741074 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:21:00.741766 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:21:00.741767 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:21:00.742291 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:21:00.742293 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:21:00.742627 (+ 334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:21:00.742628 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:21:00.742676 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:21:00.743099 (+ 423us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 05:21:00.743104 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0812 05:21:00.744739 (+ 1635us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.100.62:41525: Server oryx/127.31.100.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":76,"threads_started":1}
W20260812 05:21:00.744880 14802 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 05:21:00.740607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.100.3:46057 (local address 127.31.100.62:41525)
0812 05:21:00.740655 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0812 05:21:00.740657 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 05:21:00.740739 (+ 82us) server_negotiation.cc:407] Connection header received
0812 05:21:00.740778 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:21:00.740779 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:21:00.740790 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:21:00.740840 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:21:00.741121 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:21:00.741693 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:21:00.742355 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:21:00.742551 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:21:00.742611 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:21:00.744820 (+ 2209us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.100.3:46057: BlockingRecv error: recv got EOF from 127.31.100.3:46057 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
Aug 12 05:21:01 dist-test-slave-42z9 krb5kdc[13098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.100.1@KRBTEST.COM for oryx/127.31.100.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 05:21:01 dist-test-slave-42z9 krb5kdc[13098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.100.1@KRBTEST.COM for oryx/127.31.100.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 05:21:01.493258 14807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:21:01.488912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.62:41525 (local address 127.31.100.1:50319)
0812 05:21:01.489043 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0812 05:21:01.489054 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0812 05:21:01.489105 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:21:01.489221 (+ 116us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:21:01.489222 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:21:01.489374 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0812 05:21:01.489466 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:21:01.489469 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:21:01.490126 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:21:01.490127 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:21:01.490654 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:21:01.490656 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:21:01.491012 (+ 356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:21:01.491013 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:21:01.491047 (+ 34us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:21:01.491501 (+ 454us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 05:21:01.491506 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0812 05:21:01.493198 (+ 1692us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.100.62:41525: Server oryx/127.31.100.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":70,"threads_started":1}
W20260812 05:21:01.493363 14808 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 05:21:01.488981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.100.1:50319 (local address 127.31.100.62:41525)
0812 05:21:01.489085 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0812 05:21:01.489087 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 05:21:01.489096 (+ 9us) server_negotiation.cc:407] Connection header received
0812 05:21:01.489147 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:21:01.489147 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:21:01.489157 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:21:01.489211 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:21:01.489531 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:21:01.490054 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:21:01.490740 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:21:01.490950 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:21:01.491006 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:21:01.493286 (+ 2280us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.100.1:50319: BlockingRecv error: recv got EOF from 127.31.100.1:50319 (error 108)
Metrics: {"server-negotiator.queue_time_us":83,"thread_start_us":52,"threads_started":1}
Aug 12 05:21:01 dist-test-slave-42z9 krb5kdc[13098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.100.2@KRBTEST.COM for oryx/127.31.100.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 05:21:01 dist-test-slave-42z9 krb5kdc[13098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.100.2@KRBTEST.COM for oryx/127.31.100.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 05:21:01.626685 14810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:21:01.617866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.62:41525 (local address 127.31.100.2:42985)
0812 05:21:01.617989 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0812 05:21:01.618000 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0812 05:21:01.618066 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:21:01.618193 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:21:01.618194 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:21:01.618353 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0812 05:21:01.618404 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:21:01.618406 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:21:01.619043 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:21:01.619045 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:21:01.619598 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:21:01.619600 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:21:01.619936 (+ 336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:21:01.619937 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:21:01.619980 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:21:01.624764 (+ 4784us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 05:21:01.624769 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0812 05:21:01.626626 (+ 1857us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.100.62:41525: Server oryx/127.31.100.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":60,"threads_started":1}
W20260812 05:21:01.626791 14808 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 05:21:01.617954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.100.2:42985 (local address 127.31.100.62:41525)
0812 05:21:01.618002 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0812 05:21:01.618004 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 05:21:01.618049 (+ 45us) server_negotiation.cc:407] Connection header received
0812 05:21:01.618124 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:21:01.618125 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:21:01.618140 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:21:01.618188 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:21:01.618463 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:21:01.618960 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:21:01.619680 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:21:01.619874 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:21:01.619932 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:21:01.626728 (+ 6796us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.100.2:42985: BlockingRecv error: recv got EOF from 127.31.100.2:42985 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
Aug 12 05:21:01 dist-test-slave-42z9 krb5kdc[13098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.100.3@KRBTEST.COM for oryx/127.31.100.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 05:21:01 dist-test-slave-42z9 krb5kdc[13098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.100.3@KRBTEST.COM for oryx/127.31.100.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 05:21:01.749433 14811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:21:01.745242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.62:41525 (local address 127.31.100.3:35671)
0812 05:21:01.745351 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0812 05:21:01.745362 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0812 05:21:01.745410 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:21:01.745591 (+ 181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:21:01.745592 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:21:01.745746 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0812 05:21:01.745797 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:21:01.745799 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:21:01.746384 (+ 585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:21:01.746384 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:21:01.746923 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:21:01.746925 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:21:01.747198 (+ 273us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:21:01.747199 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:21:01.747235 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:21:01.747670 (+ 435us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 05:21:01.747675 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0812 05:21:01.749368 (+ 1693us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.100.62:41525: Server oryx/127.31.100.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":87,"thread_start_us":52,"threads_started":1}
W20260812 05:21:01.749539 14808 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 05:21:01.745307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.100.3:35671 (local address 127.31.100.62:41525)
0812 05:21:01.745385 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0812 05:21:01.745387 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 05:21:01.745412 (+ 25us) server_negotiation.cc:407] Connection header received
0812 05:21:01.745528 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:21:01.745529 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:21:01.745544 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:21:01.745585 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:21:01.745844 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:21:01.746332 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:21:01.746980 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:21:01.747152 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:21:01.747195 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:21:01.749473 (+ 2278us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.100.3:35671: BlockingRecv error: recv got EOF from 127.31.100.3:35671 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Aug 12 05:21:02 dist-test-slave-42z9 krb5kdc[13098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.100.1@KRBTEST.COM for oryx/127.31.100.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 05:21:02 dist-test-slave-42z9 krb5kdc[13098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.100.1@KRBTEST.COM for oryx/127.31.100.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 05:21:02.498270 14812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:21:02.493705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.62:41525 (local address 127.31.100.1:48065)
0812 05:21:02.493835 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0812 05:21:02.493845 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0812 05:21:02.493892 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:21:02.494013 (+ 121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:21:02.494013 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:21:02.494171 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0812 05:21:02.494221 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:21:02.494223 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:21:02.494874 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:21:02.494875 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:21:02.495599 (+ 724us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:21:02.495602 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:21:02.495946 (+ 344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:21:02.495947 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:21:02.495994 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:21:02.496454 (+ 460us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 05:21:02.496458 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0812 05:21:02.498209 (+ 1751us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.100.62:41525: Server oryx/127.31.100.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":57,"threads_started":1}
W20260812 05:21:02.498402 14818 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 05:21:02.493770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.100.1:48065 (local address 127.31.100.62:41525)
0812 05:21:02.493890 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0812 05:21:02.493891 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0812 05:21:02.493900 (+ 9us) server_negotiation.cc:407] Connection header received
0812 05:21:02.493933 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:21:02.493934 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:21:02.493944 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:21:02.494002 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:21:02.494273 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:21:02.494783 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:21:02.495665 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:21:02.495867 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:21:02.495930 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:21:02.498330 (+ 2400us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.100.1:48065: BlockingRecv error: recv got EOF from 127.31.100.1:48065 (error 108)
Metrics: {"server-negotiator.queue_time_us":94,"thread_start_us":56,"threads_started":1}
Aug 12 05:21:02 dist-test-slave-42z9 krb5kdc[13098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.100.2@KRBTEST.COM for oryx/127.31.100.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 05:21:02 dist-test-slave-42z9 krb5kdc[13098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.100.2@KRBTEST.COM for oryx/127.31.100.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 05:21:02.631448 14821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:21:02.627120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.62:41525 (local address 127.31.100.2:45285)
0812 05:21:02.627244 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0812 05:21:02.627253 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0812 05:21:02.627302 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:21:02.627419 (+ 117us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:21:02.627420 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:21:02.627582 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0812 05:21:02.627632 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:21:02.627634 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:21:02.628292 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:21:02.628293 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:21:02.628832 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:21:02.628834 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:21:02.629134 (+ 300us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:21:02.629135 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:21:02.629172 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:21:02.629687 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 05:21:02.629692 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0812 05:21:02.631384 (+ 1692us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.100.62:41525: Server oryx/127.31.100.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":59,"threads_started":1}
W20260812 05:21:02.631547 14818 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 05:21:02.627225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.100.2:45285 (local address 127.31.100.62:41525)
0812 05:21:02.627270 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0812 05:21:02.627271 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0812 05:21:02.627290 (+ 19us) server_negotiation.cc:407] Connection header received
0812 05:21:02.627355 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:21:02.627356 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:21:02.627366 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:21:02.627406 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:21:02.627716 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:21:02.628222 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:21:02.628890 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:21:02.629082 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:21:02.629128 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:21:02.631487 (+ 2359us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.100.2:45285: BlockingRecv error: recv got EOF from 127.31.100.2:45285 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
Aug 12 05:21:02 dist-test-slave-42z9 krb5kdc[13098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.100.3@KRBTEST.COM for oryx/127.31.100.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 05:21:02 dist-test-slave-42z9 krb5kdc[13098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.100.3@KRBTEST.COM for oryx/127.31.100.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 05:21:02.754601 14826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:21:02.749954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.62:41525 (local address 127.31.100.3:55567)
0812 05:21:02.750096 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0812 05:21:02.750108 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0812 05:21:02.750178 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:21:02.750312 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:21:02.750313 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:21:02.750491 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0812 05:21:02.750541 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:21:02.750543 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:21:02.751192 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:21:02.751193 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:21:02.751721 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:21:02.751723 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:21:02.752241 (+ 518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:21:02.752242 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:21:02.752312 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:21:02.752852 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 05:21:02.752857 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0812 05:21:02.754543 (+ 1686us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.100.62:41525: Server oryx/127.31.100.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":80,"threads_started":1}
W20260812 05:21:02.754688 14818 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 05:21:02.750041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.100.3:55567 (local address 127.31.100.62:41525)
0812 05:21:02.750104 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0812 05:21:02.750106 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 05:21:02.750166 (+ 60us) server_negotiation.cc:407] Connection header received
0812 05:21:02.750236 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:21:02.750236 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:21:02.750251 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:21:02.750297 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:21:02.750612 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:21:02.751118 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:21:02.751828 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:21:02.752165 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:21:02.752239 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:21:02.754624 (+ 2385us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.100.3:55567: BlockingRecv error: recv got EOF from 127.31.100.3:55567 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
Aug 12 05:21:03 dist-test-slave-42z9 krb5kdc[13098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.100.1@KRBTEST.COM for oryx/127.31.100.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 05:21:03 dist-test-slave-42z9 krb5kdc[13098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.100.1@KRBTEST.COM for oryx/127.31.100.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 05:21:03.503288 14829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:21:03.498715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.62:41525 (local address 127.31.100.1:37667)
0812 05:21:03.498849 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0812 05:21:03.498861 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0812 05:21:03.498915 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:21:03.499043 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:21:03.499044 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:21:03.499203 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0812 05:21:03.499253 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:21:03.499255 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:21:03.500084 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:21:03.500085 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:21:03.500648 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:21:03.500650 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:21:03.500985 (+ 335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:21:03.500986 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:21:03.501042 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:21:03.501472 (+ 430us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 05:21:03.501475 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0812 05:21:03.503228 (+ 1753us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.100.62:41525: Server oryx/127.31.100.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":69,"threads_started":1}
W20260812 05:21:03.503369 14832 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 05:21:03.498781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.100.1:37667 (local address 127.31.100.62:41525)
0812 05:21:03.498888 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0812 05:21:03.498890 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 05:21:03.498901 (+ 11us) server_negotiation.cc:407] Connection header received
0812 05:21:03.498966 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:21:03.498967 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:21:03.498978 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:21:03.499033 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:21:03.499305 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:21:03.500016 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:21:03.500711 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:21:03.500908 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:21:03.500951 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:21:03.503315 (+ 2364us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.100.1:37667: BlockingRecv error: recv got EOF from 127.31.100.1:37667 (error 108)
Metrics: {"server-negotiator.queue_time_us":88,"thread_start_us":53,"threads_started":1}
Aug 12 05:21:03 dist-test-slave-42z9 krb5kdc[13098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.100.2@KRBTEST.COM for oryx/127.31.100.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 05:21:03 dist-test-slave-42z9 krb5kdc[13098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.100.2@KRBTEST.COM for oryx/127.31.100.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 05:21:03.636272 14833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:21:03.631904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.62:41525 (local address 127.31.100.2:39877)
0812 05:21:03.632041 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0812 05:21:03.632052 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0812 05:21:03.632111 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:21:03.632225 (+ 114us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:21:03.632226 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:21:03.632391 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0812 05:21:03.632442 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:21:03.632443 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:21:03.633123 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:21:03.633124 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:21:03.633721 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:21:03.633724 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:21:03.634053 (+ 329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:21:03.634054 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:21:03.634105 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:21:03.634504 (+ 399us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 05:21:03.634509 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0812 05:21:03.636212 (+ 1703us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.100.62:41525: Server oryx/127.31.100.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":80,"threads_started":1}
W20260812 05:21:03.636366 14832 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 05:21:03.631983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.100.2:39877 (local address 127.31.100.62:41525)
0812 05:21:03.632069 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0812 05:21:03.632071 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 05:21:03.632097 (+ 26us) server_negotiation.cc:407] Connection header received
0812 05:21:03.632165 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:21:03.632166 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:21:03.632176 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:21:03.632220 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:21:03.632518 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:21:03.633037 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:21:03.633787 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:21:03.633979 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:21:03.634026 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:21:03.636307 (+ 2281us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.100.2:39877: BlockingRecv error: recv got EOF from 127.31.100.2:39877 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Aug 12 05:21:03 dist-test-slave-42z9 krb5kdc[13098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.100.3@KRBTEST.COM for oryx/127.31.100.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 05:21:03 dist-test-slave-42z9 krb5kdc[13098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.100.3@KRBTEST.COM for oryx/127.31.100.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 05:21:03.759859 14835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:21:03.755090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.62:41525 (local address 127.31.100.3:40697)
0812 05:21:03.755223 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0812 05:21:03.755235 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0812 05:21:03.755301 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:21:03.755468 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:21:03.755469 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:21:03.755636 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0812 05:21:03.755684 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:21:03.755686 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:21:03.756323 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:21:03.756324 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:21:03.756851 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:21:03.756853 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:21:03.757173 (+ 320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:21:03.757174 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:21:03.757230 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:21:03.757725 (+ 495us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 05:21:03.757730 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0812 05:21:03.759780 (+ 2050us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.100.62:41525: Server oryx/127.31.100.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":66,"threads_started":1}
W20260812 05:21:03.759943 14832 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 05:21:03.755173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.100.3:40697 (local address 127.31.100.62:41525)
0812 05:21:03.755226 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0812 05:21:03.755228 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 05:21:03.755282 (+ 54us) server_negotiation.cc:407] Connection header received
0812 05:21:03.755359 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:21:03.755360 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:21:03.755373 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:21:03.755416 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:21:03.755755 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:21:03.756253 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:21:03.756930 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:21:03.757110 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:21:03.757164 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:21:03.759876 (+ 2712us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.100.3:40697: BlockingRecv error: recv got EOF from 127.31.100.3:40697 (error 108)
Metrics: {"server-negotiator.queue_time_us":29}
Aug 12 05:21:04 dist-test-slave-42z9 krb5kdc[13098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.100.1@KRBTEST.COM for oryx/127.31.100.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 05:21:04 dist-test-slave-42z9 krb5kdc[13098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.100.1@KRBTEST.COM for oryx/127.31.100.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 05:21:04.507865 14837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:21:04.503711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.62:41525 (local address 127.31.100.1:47823)
0812 05:21:04.503836 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0812 05:21:04.503847 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0812 05:21:04.503899 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:21:04.504009 (+ 110us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:21:04.504010 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:21:04.504170 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0812 05:21:04.504221 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:21:04.504223 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:21:04.504847 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:21:04.504848 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:21:04.505398 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:21:04.505401 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:21:04.505731 (+ 330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:21:04.505731 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:21:04.505770 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:21:04.506160 (+ 390us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 05:21:04.506164 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0812 05:21:04.507797 (+ 1633us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.100.62:41525: Server oryx/127.31.100.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":60,"threads_started":1}
W20260812 05:21:04.507961 14839 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 05:21:04.503760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.100.1:47823 (local address 127.31.100.62:41525)
0812 05:21:04.503898 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0812 05:21:04.503900 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 05:21:04.503908 (+ 8us) server_negotiation.cc:407] Connection header received
0812 05:21:04.503932 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:21:04.503933 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:21:04.503943 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:21:04.504000 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:21:04.504271 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:21:04.504772 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:21:04.505489 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:21:04.505682 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:21:04.505726 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:21:04.507893 (+ 2167us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.100.1:47823: BlockingRecv error: recv got EOF from 127.31.100.1:47823 (error 108)
Metrics: {"server-negotiator.queue_time_us":107,"thread_start_us":76,"threads_started":1}
Aug 12 05:21:04 dist-test-slave-42z9 krb5kdc[13098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.100.2@KRBTEST.COM for oryx/127.31.100.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 05:21:04 dist-test-slave-42z9 krb5kdc[13098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.100.2@KRBTEST.COM for oryx/127.31.100.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 05:21:04.641407 14841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:21:04.636785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.62:41525 (local address 127.31.100.2:32925)
0812 05:21:04.636943 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0812 05:21:04.636954 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0812 05:21:04.637013 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:21:04.637136 (+ 123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:21:04.637137 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:21:04.637304 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0812 05:21:04.637354 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:21:04.637356 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:21:04.637999 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:21:04.638000 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:21:04.638545 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:21:04.638547 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:21:04.638914 (+ 367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:21:04.638915 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:21:04.638970 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:21:04.639421 (+ 451us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 05:21:04.639426 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0812 05:21:04.641348 (+ 1922us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.100.62:41525: Server oryx/127.31.100.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":92,"threads_started":1}
W20260812 05:21:04.641537 14839 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 05:21:04.636886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.100.2:32925 (local address 127.31.100.62:41525)
0812 05:21:04.636964 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0812 05:21:04.636966 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 05:21:04.637009 (+ 43us) server_negotiation.cc:407] Connection header received
0812 05:21:04.637057 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:21:04.637057 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:21:04.637069 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:21:04.637125 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:21:04.637410 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:21:04.637942 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:21:04.638629 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:21:04.638831 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:21:04.638909 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:21:04.641475 (+ 2566us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.100.2:32925: BlockingRecv error: recv got EOF from 127.31.100.2:32925 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Aug 12 05:21:04 dist-test-slave-42z9 krb5kdc[13098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.100.3@KRBTEST.COM for oryx/127.31.100.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 05:21:04 dist-test-slave-42z9 krb5kdc[13098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.100.3@KRBTEST.COM for oryx/127.31.100.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 05:21:04.764727 14843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:21:04.760346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.62:41525 (local address 127.31.100.3:35847)
0812 05:21:04.760508 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0812 05:21:04.760520 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0812 05:21:04.760595 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:21:04.760729 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:21:04.760730 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:21:04.760905 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0812 05:21:04.760956 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:21:04.760957 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:21:04.761656 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:21:04.761657 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:21:04.762182 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:21:04.762184 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:21:04.762503 (+ 319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:21:04.762504 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:21:04.762567 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:21:04.762958 (+ 391us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 05:21:04.762962 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0812 05:21:04.764669 (+ 1707us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.100.62:41525: Server oryx/127.31.100.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":98,"threads_started":1}
W20260812 05:21:04.764822 14839 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 05:21:04.760424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.100.3:35847 (local address 127.31.100.62:41525)
0812 05:21:04.760497 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0812 05:21:04.760500 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 05:21:04.760583 (+ 83us) server_negotiation.cc:407] Connection header received
0812 05:21:04.760656 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:21:04.760657 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:21:04.760670 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:21:04.760718 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:21:04.761021 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:21:04.761575 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:21:04.762250 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:21:04.762443 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:21:04.762498 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:21:04.764760 (+ 2262us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.100.3:35847: BlockingRecv error: recv got EOF from 127.31.100.3:35847 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Aug 12 05:21:05 dist-test-slave-42z9 krb5kdc[13098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.100.1@KRBTEST.COM for oryx/127.31.100.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 05:21:05 dist-test-slave-42z9 krb5kdc[13098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.100.1@KRBTEST.COM for oryx/127.31.100.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 05:21:05.512907 14844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:21:05.508313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.62:41525 (local address 127.31.100.1:52681)
0812 05:21:05.508447 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0812 05:21:05.508460 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 05:21:05.508522 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:21:05.508725 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:21:05.508726 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:21:05.508882 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0812 05:21:05.508933 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:21:05.508935 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:21:05.509722 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:21:05.509723 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:21:05.510290 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:21:05.510292 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:21:05.510639 (+ 347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:21:05.510640 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:21:05.510690 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:21:05.511113 (+ 423us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 05:21:05.511117 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0812 05:21:05.512843 (+ 1726us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.100.62:41525: Server oryx/127.31.100.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":68,"threads_started":1}
W20260812 05:21:05.513022 14845 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 05:21:05.508437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.100.1:52681 (local address 127.31.100.62:41525)
0812 05:21:05.508609 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0812 05:21:05.508611 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 05:21:05.508619 (+ 8us) server_negotiation.cc:407] Connection header received
0812 05:21:05.508640 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:21:05.508640 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:21:05.508650 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:21:05.508714 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:21:05.509036 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:21:05.509653 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:21:05.510356 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:21:05.510576 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:21:05.510638 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:21:05.512927 (+ 2289us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.100.1:52681: BlockingRecv error: recv got EOF from 127.31.100.1:52681 (error 108)
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":93,"threads_started":1}
Aug 12 05:21:05 dist-test-slave-42z9 krb5kdc[13098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.100.2@KRBTEST.COM for oryx/127.31.100.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 05:21:05 dist-test-slave-42z9 krb5kdc[13098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.100.2@KRBTEST.COM for oryx/127.31.100.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 05:21:05.648767 14848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:21:05.641894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.62:41525 (local address 127.31.100.2:49141)
0812 05:21:05.644053 (+ 2159us) negotiation.cc:107] Waiting for socket to connect
0812 05:21:05.644064 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0812 05:21:05.644127 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:21:05.644255 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:21:05.644256 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:21:05.644448 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0812 05:21:05.644503 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:21:05.644505 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:21:05.645298 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:21:05.645300 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:21:05.645897 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:21:05.645900 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:21:05.646396 (+ 496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:21:05.646397 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:21:05.646464 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:21:05.646959 (+ 495us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 05:21:05.646965 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0812 05:21:05.648699 (+ 1734us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.100.62:41525: Server oryx/127.31.100.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":2140,"thread_start_us":2077,"threads_started":1}
W20260812 05:21:05.648859 14845 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 05:21:05.641957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.100.2:49141 (local address 127.31.100.62:41525)
0812 05:21:05.642010 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0812 05:21:05.642012 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 05:21:05.644126 (+ 2114us) server_negotiation.cc:407] Connection header received
0812 05:21:05.644171 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:21:05.644172 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:21:05.644185 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:21:05.644240 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:21:05.644564 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:21:05.645211 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:21:05.646003 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:21:05.646272 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:21:05.646359 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:21:05.648799 (+ 2440us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.100.2:49141: BlockingRecv error: recv got EOF from 127.31.100.2:49141 (error 108)
Metrics: {"server-negotiator.queue_time_us":27}
Aug 12 05:21:05 dist-test-slave-42z9 krb5kdc[13098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.100.3@KRBTEST.COM for oryx/127.31.100.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 05:21:05 dist-test-slave-42z9 krb5kdc[13098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.100.3@KRBTEST.COM for oryx/127.31.100.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 05:21:05.769647 14849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:21:05.765168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.62:41525 (local address 127.31.100.3:48265)
0812 05:21:05.765310 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0812 05:21:05.765321 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0812 05:21:05.765382 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:21:05.765548 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:21:05.765549 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:21:05.765731 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0812 05:21:05.765779 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:21:05.765780 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:21:05.766413 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:21:05.766415 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:21:05.766958 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:21:05.766960 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:21:05.767282 (+ 322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:21:05.767283 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:21:05.767339 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:21:05.767755 (+ 416us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 05:21:05.767759 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0812 05:21:05.769579 (+ 1820us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.100.62:41525: Server oryx/127.31.100.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":79,"threads_started":1}
W20260812 05:21:05.769737 14845 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 05:21:05.765215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.100.3:48265 (local address 127.31.100.62:41525)
0812 05:21:05.765257 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0812 05:21:05.765259 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 05:21:05.765373 (+ 114us) server_negotiation.cc:407] Connection header received
0812 05:21:05.765473 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:21:05.765474 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:21:05.765488 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:21:05.765533 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:21:05.765851 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:21:05.766342 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:21:05.767036 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:21:05.767223 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:21:05.767276 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:21:05.769676 (+ 2400us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.100.3:48265: BlockingRecv error: recv got EOF from 127.31.100.3:48265 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
Aug 12 05:21:06 dist-test-slave-42z9 krb5kdc[13098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.100.1@KRBTEST.COM for oryx/127.31.100.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 05:21:06 dist-test-slave-42z9 krb5kdc[13098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.100.1@KRBTEST.COM for oryx/127.31.100.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 05:21:06.517772 14850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:21:06.513339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.62:41525 (local address 127.31.100.1:56025)
0812 05:21:06.513498 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0812 05:21:06.513510 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0812 05:21:06.513562 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:21:06.513677 (+ 115us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:21:06.513678 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:21:06.513837 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0812 05:21:06.513888 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:21:06.513889 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:21:06.514519 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:21:06.514520 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:21:06.515048 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:21:06.515049 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:21:06.515396 (+ 347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:21:06.515397 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:21:06.515468 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:21:06.515886 (+ 418us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 05:21:06.515891 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0812 05:21:06.517694 (+ 1803us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.100.62:41525: Server oryx/127.31.100.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":74,"threads_started":1}
W20260812 05:21:06.517853 14852 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 05:21:06.513430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.100.1:56025 (local address 127.31.100.62:41525)
0812 05:21:06.513569 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0812 05:21:06.513571 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 05:21:06.513578 (+ 7us) server_negotiation.cc:407] Connection header received
0812 05:21:06.513600 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:21:06.513600 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:21:06.513610 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:21:06.513666 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:21:06.513940 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:21:06.514430 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:21:06.515132 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:21:06.515332 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:21:06.515385 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:21:06.517790 (+ 2405us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.100.1:56025: BlockingRecv error: recv got EOF from 127.31.100.1:56025 (error 108)
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":57,"threads_started":1}
Aug 12 05:21:06 dist-test-slave-42z9 krb5kdc[13098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.100.2@KRBTEST.COM for oryx/127.31.100.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 05:21:06 dist-test-slave-42z9 krb5kdc[13098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.100.2@KRBTEST.COM for oryx/127.31.100.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 05:21:06.656970 14853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:21:06.649243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.62:41525 (local address 127.31.100.2:53583)
0812 05:21:06.649419 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0812 05:21:06.649430 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0812 05:21:06.649534 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:21:06.649671 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:21:06.649672 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:21:06.649834 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0812 05:21:06.649886 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:21:06.649888 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:21:06.650533 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:21:06.650534 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:21:06.651053 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:21:06.651054 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:21:06.651388 (+ 334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:21:06.651389 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:21:06.651445 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:21:06.655041 (+ 3596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 05:21:06.655046 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0812 05:21:06.656892 (+ 1846us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.100.62:41525: Server oryx/127.31.100.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":89,"threads_started":1}
W20260812 05:21:06.657055 14852 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 05:21:06.649320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.100.2:53583 (local address 127.31.100.62:41525)
0812 05:21:06.649383 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0812 05:21:06.649386 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 05:21:06.649522 (+ 136us) server_negotiation.cc:407] Connection header received
0812 05:21:06.649587 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:21:06.649588 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:21:06.649600 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:21:06.649654 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:21:06.649962 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:21:06.650477 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:21:06.651115 (+ 638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:21:06.651304 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:21:06.651352 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:21:06.657000 (+ 5648us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.100.2:53583: BlockingRecv error: recv got EOF from 127.31.100.2:53583 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
Aug 12 05:21:06 dist-test-slave-42z9 krb5kdc[13098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.100.3@KRBTEST.COM for oryx/127.31.100.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 05:21:06 dist-test-slave-42z9 krb5kdc[13098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.100.3@KRBTEST.COM for oryx/127.31.100.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 05:21:06.774425 14855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:21:06.770080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.62:41525 (local address 127.31.100.3:34313)
0812 05:21:06.770202 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0812 05:21:06.770213 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0812 05:21:06.770283 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:21:06.770415 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:21:06.770416 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:21:06.770587 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0812 05:21:06.770635 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:21:06.770637 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:21:06.771279 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:21:06.771281 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:21:06.771830 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:21:06.771832 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:21:06.772162 (+ 330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:21:06.772163 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:21:06.772218 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:21:06.772638 (+ 420us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 05:21:06.772643 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0812 05:21:06.774337 (+ 1694us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.100.62:41525: Server oryx/127.31.100.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":59,"threads_started":1}
W20260812 05:21:06.774490 14852 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 05:21:06.770144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.100.3:34313 (local address 127.31.100.62:41525)
0812 05:21:06.770206 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0812 05:21:06.770208 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 05:21:06.770270 (+ 62us) server_negotiation.cc:407] Connection header received
0812 05:21:06.770336 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:21:06.770337 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:21:06.770351 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:21:06.770403 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:21:06.770701 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:21:06.771202 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:21:06.771914 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:21:06.772103 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:21:06.772157 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:21:06.774436 (+ 2279us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.100.3:34313: BlockingRecv error: recv got EOF from 127.31.100.3:34313 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
Aug 12 05:21:07 dist-test-slave-42z9 krb5kdc[13098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.100.1@KRBTEST.COM for oryx/127.31.100.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 05:21:07 dist-test-slave-42z9 krb5kdc[13098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.100.1@KRBTEST.COM for oryx/127.31.100.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 05:21:07.522591 14858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:21:07.518206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.62:41525 (local address 127.31.100.1:54467)
0812 05:21:07.518327 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0812 05:21:07.518337 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0812 05:21:07.518389 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:21:07.518514 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:21:07.518514 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:21:07.518670 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0812 05:21:07.518721 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:21:07.518722 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:21:07.519328 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:21:07.519328 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:21:07.519852 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:21:07.519853 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:21:07.520178 (+ 325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:21:07.520179 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:21:07.520233 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:21:07.520730 (+ 497us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 05:21:07.520734 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0812 05:21:07.522511 (+ 1777us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.100.62:41525: Server oryx/127.31.100.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":65,"threads_started":1}
W20260812 05:21:07.522670 14860 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 05:21:07.518274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.100.1:54467 (local address 127.31.100.62:41525)
0812 05:21:07.518378 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0812 05:21:07.518380 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 05:21:07.518388 (+ 8us) server_negotiation.cc:407] Connection header received
0812 05:21:07.518447 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:21:07.518447 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:21:07.518459 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:21:07.518504 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:21:07.518768 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:21:07.519276 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:21:07.519910 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:21:07.520106 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:21:07.520155 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:21:07.522610 (+ 2455us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.100.1:54467: BlockingRecv error: recv got EOF from 127.31.100.1:54467 (error 108)
Metrics: {"server-negotiator.queue_time_us":85,"thread_start_us":50,"threads_started":1}
Aug 12 05:21:07 dist-test-slave-42z9 krb5kdc[13098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.100.2@KRBTEST.COM for oryx/127.31.100.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 05:21:07 dist-test-slave-42z9 krb5kdc[13098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.100.2@KRBTEST.COM for oryx/127.31.100.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 05:21:07.662061 14861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:21:07.657496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.62:41525 (local address 127.31.100.2:60863)
0812 05:21:07.657628 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0812 05:21:07.657638 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0812 05:21:07.657706 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:21:07.657828 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:21:07.657829 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:21:07.658013 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0812 05:21:07.658061 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:21:07.658063 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:21:07.658775 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:21:07.658776 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:21:07.659299 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:21:07.659301 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:21:07.659738 (+ 437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:21:07.659740 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:21:07.659796 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:21:07.660233 (+ 437us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 05:21:07.660238 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0812 05:21:07.661966 (+ 1728us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.100.62:41525: Server oryx/127.31.100.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":71,"threads_started":1}
W20260812 05:21:07.662119 14860 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 05:21:07.657514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.100.2:60863 (local address 127.31.100.62:41525)
0812 05:21:07.657585 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0812 05:21:07.657588 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 05:21:07.657697 (+ 109us) server_negotiation.cc:407] Connection header received
0812 05:21:07.657763 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:21:07.657764 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:21:07.657776 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:21:07.657823 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:21:07.658135 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:21:07.658694 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:21:07.659386 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:21:07.659666 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:21:07.659718 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:21:07.662065 (+ 2347us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.100.2:60863: BlockingRecv error: recv got EOF from 127.31.100.2:60863 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
Aug 12 05:21:07 dist-test-slave-42z9 krb5kdc[13098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.100.3@KRBTEST.COM for oryx/127.31.100.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 05:21:07 dist-test-slave-42z9 krb5kdc[13098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.100.3@KRBTEST.COM for oryx/127.31.100.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 05:21:07.779444 14863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:21:07.774957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.62:41525 (local address 127.31.100.3:53709)
0812 05:21:07.775121 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0812 05:21:07.775132 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0812 05:21:07.775190 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:21:07.775308 (+ 118us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:21:07.775309 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:21:07.775481 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0812 05:21:07.775533 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:21:07.775534 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:21:07.776132 (+ 598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:21:07.776133 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:21:07.776707 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:21:07.776710 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:21:07.777132 (+ 422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:21:07.777134 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:21:07.777193 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:21:07.777648 (+ 455us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 05:21:07.777651 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0812 05:21:07.779342 (+ 1691us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.100.62:41525: Server oryx/127.31.100.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":83,"threads_started":1}
W20260812 05:21:07.779506 14860 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 05:21:07.775092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.100.3:53709 (local address 127.31.100.62:41525)
0812 05:21:07.775147 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0812 05:21:07.775148 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0812 05:21:07.775176 (+ 28us) server_negotiation.cc:407] Connection header received
0812 05:21:07.775237 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:21:07.775238 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:21:07.775248 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:21:07.775297 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:21:07.775583 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:21:07.776072 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:21:07.776781 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:21:07.777060 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:21:07.777126 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:21:07.779451 (+ 2325us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.100.3:53709: BlockingRecv error: recv got EOF from 127.31.100.3:53709 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
Aug 12 05:21:08 dist-test-slave-42z9 krb5kdc[13098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.100.1@KRBTEST.COM for oryx/127.31.100.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 05:21:08 dist-test-slave-42z9 krb5kdc[13098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.100.1@KRBTEST.COM for oryx/127.31.100.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 05:21:08.527364 14864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:21:08.523029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.62:41525 (local address 127.31.100.1:47047)
0812 05:21:08.523154 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0812 05:21:08.523164 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0812 05:21:08.523215 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:21:08.523336 (+ 121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:21:08.523337 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:21:08.523500 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0812 05:21:08.523550 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:21:08.523552 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:21:08.524191 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:21:08.524191 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:21:08.524726 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:21:08.524728 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:21:08.525027 (+ 299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:21:08.525027 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:21:08.525063 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:21:08.525497 (+ 434us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 05:21:08.525501 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0812 05:21:08.527259 (+ 1758us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.100.62:41525: Server oryx/127.31.100.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":69,"threads_started":1}
W20260812 05:21:08.527457 14865 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 05:21:08.523094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.100.1:47047 (local address 127.31.100.62:41525)
0812 05:21:08.523194 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0812 05:21:08.523196 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 05:21:08.523206 (+ 10us) server_negotiation.cc:407] Connection header received
0812 05:21:08.523262 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:21:08.523262 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:21:08.523273 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:21:08.523327 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:21:08.523611 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:21:08.524122 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:21:08.524781 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:21:08.524977 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:21:08.525024 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:21:08.527355 (+ 2331us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.100.1:47047: BlockingRecv error: recv got EOF from 127.31.100.1:47047 (error 108)
Metrics: {"server-negotiator.queue_time_us":80,"thread_start_us":49,"threads_started":1}
Aug 12 05:21:08 dist-test-slave-42z9 krb5kdc[13098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.100.2@KRBTEST.COM for oryx/127.31.100.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 05:21:08 dist-test-slave-42z9 krb5kdc[13098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.100.2@KRBTEST.COM for oryx/127.31.100.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 05:21:08.667191 14866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:21:08.662515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.62:41525 (local address 127.31.100.2:58693)
0812 05:21:08.662654 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0812 05:21:08.662667 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 05:21:08.662737 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:21:08.662862 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:21:08.662863 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:21:08.663049 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0812 05:21:08.663095 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:21:08.663097 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:21:08.663745 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:21:08.663746 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:21:08.664297 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:21:08.664300 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:21:08.664635 (+ 335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:21:08.664636 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:21:08.664693 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:21:08.665156 (+ 463us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 05:21:08.665161 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0812 05:21:08.667100 (+ 1939us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.100.62:41525: Server oryx/127.31.100.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":71,"threads_started":1}
W20260812 05:21:08.667268 14865 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 05:21:08.662604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.100.2:58693 (local address 127.31.100.62:41525)
0812 05:21:08.662662 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0812 05:21:08.662665 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 05:21:08.662717 (+ 52us) server_negotiation.cc:407] Connection header received
0812 05:21:08.662789 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:21:08.662790 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:21:08.662802 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:21:08.662848 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:21:08.663166 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:21:08.663669 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:21:08.664377 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:21:08.664573 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:21:08.664630 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:21:08.667213 (+ 2583us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.100.2:58693: BlockingRecv error: recv got EOF from 127.31.100.2:58693 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
Aug 12 05:21:08 dist-test-slave-42z9 krb5kdc[13098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.100.3@KRBTEST.COM for oryx/127.31.100.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 05:21:08 dist-test-slave-42z9 krb5kdc[13098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.100.3@KRBTEST.COM for oryx/127.31.100.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 05:21:08.784276 14867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:21:08.779965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.62:41525 (local address 127.31.100.3:38313)
0812 05:21:08.780116 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0812 05:21:08.780126 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0812 05:21:08.780169 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:21:08.780274 (+ 105us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:21:08.780274 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:21:08.780443 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0812 05:21:08.780500 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:21:08.780502 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:21:08.781112 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:21:08.781112 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:21:08.781688 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:21:08.781691 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:21:08.782015 (+ 324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:21:08.782016 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:21:08.782053 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:21:08.782520 (+ 467us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 05:21:08.782526 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0812 05:21:08.784208 (+ 1682us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.100.62:41525: Server oryx/127.31.100.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":92,"threads_started":1}
W20260812 05:21:08.784360 14865 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 05:21:08.780065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.100.3:38313 (local address 127.31.100.62:41525)
0812 05:21:08.780146 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0812 05:21:08.780148 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 05:21:08.780178 (+ 30us) server_negotiation.cc:407] Connection header received
0812 05:21:08.780217 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:21:08.780217 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:21:08.780229 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:21:08.780269 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:21:08.780557 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:21:08.781057 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:21:08.781763 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:21:08.781959 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:21:08.782014 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:21:08.784297 (+ 2283us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.100.3:38313: BlockingRecv error: recv got EOF from 127.31.100.3:38313 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Aug 12 05:21:09 dist-test-slave-42z9 krb5kdc[13098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.100.1@KRBTEST.COM for oryx/127.31.100.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 05:21:09 dist-test-slave-42z9 krb5kdc[13098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.100.1@KRBTEST.COM for oryx/127.31.100.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 05:21:09.532353 14870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:21:09.527858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.62:41525 (local address 127.31.100.1:59397)
0812 05:21:09.527988 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0812 05:21:09.527999 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0812 05:21:09.528052 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:21:09.528166 (+ 114us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:21:09.528167 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:21:09.528333 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0812 05:21:09.528389 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:21:09.528392 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:21:09.529043 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:21:09.529044 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:21:09.529707 (+ 663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:21:09.529709 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:21:09.530041 (+ 332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:21:09.530042 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:21:09.530081 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:21:09.530599 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 05:21:09.530603 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0812 05:21:09.532293 (+ 1690us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.100.62:41525: Server oryx/127.31.100.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":69,"threads_started":1}
W20260812 05:21:09.532488 14871 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 05:21:09.527939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.100.1:59397 (local address 127.31.100.62:41525)
0812 05:21:09.528057 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0812 05:21:09.528059 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 05:21:09.528068 (+ 9us) server_negotiation.cc:407] Connection header received
0812 05:21:09.528091 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:21:09.528092 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:21:09.528101 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:21:09.528157 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:21:09.528457 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:21:09.528973 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:21:09.529777 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:21:09.529984 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:21:09.530046 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:21:09.532380 (+ 2334us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.100.1:59397: BlockingRecv error: recv got EOF from 127.31.100.1:59397 (error 108)
Metrics: {"server-negotiator.queue_time_us":88,"thread_start_us":51,"threads_started":1}
Aug 12 05:21:09 dist-test-slave-42z9 krb5kdc[13098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.100.2@KRBTEST.COM for oryx/127.31.100.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 05:21:09 dist-test-slave-42z9 krb5kdc[13098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.100.2@KRBTEST.COM for oryx/127.31.100.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 05:21:09.672501 14872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:21:09.667693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.62:41525 (local address 127.31.100.2:53615)
0812 05:21:09.667841 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0812 05:21:09.667852 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0812 05:21:09.667916 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:21:09.668051 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:21:09.668052 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:21:09.668222 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0812 05:21:09.668270 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:21:09.668271 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:21:09.668932 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:21:09.668932 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:21:09.669535 (+ 603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:21:09.669538 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:21:09.669919 (+ 381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:21:09.669920 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:21:09.669976 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:21:09.670403 (+ 427us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 05:21:09.670407 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0812 05:21:09.672440 (+ 2033us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.100.62:41525: Server oryx/127.31.100.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":73,"threads_started":1}
W20260812 05:21:09.672606 14871 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 05:21:09.667764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.100.2:53615 (local address 127.31.100.62:41525)
0812 05:21:09.667816 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0812 05:21:09.667819 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 05:21:09.667938 (+ 119us) server_negotiation.cc:407] Connection header received
0812 05:21:09.667970 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:21:09.667971 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:21:09.667983 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:21:09.668039 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:21:09.668334 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:21:09.668857 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:21:09.669622 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:21:09.669826 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:21:09.669887 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:21:09.672558 (+ 2671us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.100.2:53615: BlockingRecv error: recv got EOF from 127.31.100.2:53615 (error 108)
Metrics: {"server-negotiator.queue_time_us":26}
Aug 12 05:21:09 dist-test-slave-42z9 krb5kdc[13098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.100.3@KRBTEST.COM for oryx/127.31.100.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 05:21:09 dist-test-slave-42z9 krb5kdc[13098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.100.3@KRBTEST.COM for oryx/127.31.100.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 05:21:09.789211 14873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:21:09.784740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.62:41525 (local address 127.31.100.3:34077)
0812 05:21:09.784903 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0812 05:21:09.784914 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0812 05:21:09.784988 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:21:09.785111 (+ 123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:21:09.785112 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:21:09.785294 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0812 05:21:09.785340 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:21:09.785342 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:21:09.786027 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:21:09.786028 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:21:09.786554 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:21:09.786556 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:21:09.786869 (+ 313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:21:09.786870 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:21:09.786932 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:21:09.787359 (+ 427us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 05:21:09.787363 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0812 05:21:09.789135 (+ 1772us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.100.62:41525: Server oryx/127.31.100.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":81,"threads_started":1}
W20260812 05:21:09.789283 14871 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 05:21:09.784858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.100.3:34077 (local address 127.31.100.62:41525)
0812 05:21:09.784908 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0812 05:21:09.784911 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 05:21:09.784982 (+ 71us) server_negotiation.cc:407] Connection header received
0812 05:21:09.785040 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:21:09.785041 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:21:09.785055 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:21:09.785099 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:21:09.785416 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:21:09.785951 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:21:09.786627 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:21:09.786810 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:21:09.786865 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:21:09.789230 (+ 2365us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.100.3:34077: BlockingRecv error: recv got EOF from 127.31.100.3:34077 (error 108)
Metrics: {"server-negotiator.queue_time_us":26}
Aug 12 05:21:10 dist-test-slave-42z9 krb5kdc[13098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.100.1@KRBTEST.COM for oryx/127.31.100.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 05:21:10 dist-test-slave-42z9 krb5kdc[13098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.100.1@KRBTEST.COM for oryx/127.31.100.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 05:21:10.537161 14874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:21:10.532791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.62:41525 (local address 127.31.100.1:58537)
0812 05:21:10.532936 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0812 05:21:10.532947 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0812 05:21:10.533009 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:21:10.533163 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:21:10.533164 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:21:10.533341 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0812 05:21:10.533392 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:21:10.533394 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:21:10.534045 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:21:10.534047 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:21:10.534583 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:21:10.534585 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:21:10.534921 (+ 336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:21:10.534922 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:21:10.534959 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:21:10.535378 (+ 419us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 05:21:10.535382 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0812 05:21:10.537073 (+ 1691us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.100.62:41525: Server oryx/127.31.100.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":64,"threads_started":1}
W20260812 05:21:10.537245 14876 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 05:21:10.532848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.100.1:58537 (local address 127.31.100.62:41525)
0812 05:21:10.532969 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0812 05:21:10.532972 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 05:21:10.532998 (+ 26us) server_negotiation.cc:407] Connection header received
0812 05:21:10.533069 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:21:10.533070 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:21:10.533080 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:21:10.533130 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:21:10.533473 (+ 343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:21:10.533985 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:21:10.534656 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:21:10.534858 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:21:10.534914 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:21:10.537176 (+ 2262us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.100.1:58537: BlockingRecv error: recv got EOF from 127.31.100.1:58537 (error 108)
Metrics: {"server-negotiator.queue_time_us":104,"thread_start_us":66,"threads_started":1}
Aug 12 05:21:10 dist-test-slave-42z9 krb5kdc[13098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.100.2@KRBTEST.COM for oryx/127.31.100.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 05:21:10 dist-test-slave-42z9 krb5kdc[13098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.100.2@KRBTEST.COM for oryx/127.31.100.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 05:21:10.677371 14877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:21:10.673034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.62:41525 (local address 127.31.100.2:60709)
0812 05:21:10.673172 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0812 05:21:10.673183 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0812 05:21:10.673232 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:21:10.673367 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:21:10.673368 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:21:10.673578 (+ 210us) client_negotiation.cc:189] Negotiated authn=SASL
0812 05:21:10.673628 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:21:10.673630 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:21:10.674237 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:21:10.674237 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:21:10.674773 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:21:10.674775 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:21:10.675059 (+ 284us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:21:10.675059 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:21:10.675096 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:21:10.675532 (+ 436us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 05:21:10.675537 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0812 05:21:10.677301 (+ 1764us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.100.62:41525: Server oryx/127.31.100.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":59,"threads_started":1}
W20260812 05:21:10.677467 14876 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 05:21:10.673152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.100.2:60709 (local address 127.31.100.62:41525)
0812 05:21:10.673208 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0812 05:21:10.673210 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 05:21:10.673220 (+ 10us) server_negotiation.cc:407] Connection header received
0812 05:21:10.673290 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:21:10.673291 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:21:10.673300 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:21:10.673352 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:21:10.673684 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:21:10.674178 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:21:10.674828 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:21:10.675010 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:21:10.675055 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:21:10.677389 (+ 2334us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.100.2:60709: BlockingRecv error: recv got EOF from 127.31.100.2:60709 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
Aug 12 05:21:10 dist-test-slave-42z9 krb5kdc[13098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.100.3@KRBTEST.COM for oryx/127.31.100.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 05:21:10 dist-test-slave-42z9 krb5kdc[13098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.100.3@KRBTEST.COM for oryx/127.31.100.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 05:21:10.794416 14878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:21:10.789708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.62:41525 (local address 127.31.100.3:34049)
0812 05:21:10.789861 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0812 05:21:10.789872 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0812 05:21:10.789942 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:21:10.790079 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:21:10.790080 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:21:10.790267 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0812 05:21:10.790310 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:21:10.790312 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:21:10.790992 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:21:10.790993 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:21:10.791547 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:21:10.791549 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:21:10.791874 (+ 325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:21:10.791875 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:21:10.791931 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:21:10.792369 (+ 438us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 05:21:10.792375 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0812 05:21:10.794343 (+ 1968us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.100.62:41525: Server oryx/127.31.100.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":74,"threads_started":1}
W20260812 05:21:10.794517 14876 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 05:21:10.789780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.100.3:34049 (local address 127.31.100.62:41525)
0812 05:21:10.789836 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0812 05:21:10.789838 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 05:21:10.789933 (+ 95us) server_negotiation.cc:407] Connection header received
0812 05:21:10.790005 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:21:10.790006 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:21:10.790019 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:21:10.790068 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:21:10.790379 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:21:10.790919 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:21:10.791627 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:21:10.791814 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:21:10.791868 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:21:10.794456 (+ 2588us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.100.3:34049: BlockingRecv error: recv got EOF from 127.31.100.3:34049 (error 108)
Metrics: {"server-negotiator.queue_time_us":26}
Aug 12 05:21:11 dist-test-slave-42z9 krb5kdc[13098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.100.1@KRBTEST.COM for oryx/127.31.100.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 05:21:11 dist-test-slave-42z9 krb5kdc[13098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.100.1@KRBTEST.COM for oryx/127.31.100.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 05:21:11.542178 14879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:21:11.537663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.62:41525 (local address 127.31.100.1:33263)
0812 05:21:11.537794 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0812 05:21:11.537805 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0812 05:21:11.537863 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:21:11.538007 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:21:11.538007 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:21:11.538174 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0812 05:21:11.538226 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:21:11.538228 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:21:11.538845 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:21:11.538846 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:21:11.539457 (+ 611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:21:11.539460 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:21:11.539777 (+ 317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:21:11.539778 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:21:11.539816 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:21:11.540218 (+ 402us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 05:21:11.540223 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0812 05:21:11.542103 (+ 1880us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.100.62:41525: Server oryx/127.31.100.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":69,"threads_started":1}
W20260812 05:21:11.542259 14880 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 05:21:11.537727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.100.1:33263 (local address 127.31.100.62:41525)
0812 05:21:11.537840 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0812 05:21:11.537842 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 05:21:11.537852 (+ 10us) server_negotiation.cc:407] Connection header received
0812 05:21:11.537912 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:21:11.537912 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:21:11.537923 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:21:11.537995 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:21:11.538275 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:21:11.538776 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:21:11.539518 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:21:11.539725 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:21:11.539773 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:21:11.542198 (+ 2425us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.100.1:33263: BlockingRecv error: recv got EOF from 127.31.100.1:33263 (error 108)
Metrics: {"server-negotiator.queue_time_us":92,"thread_start_us":55,"threads_started":1}
Aug 12 05:21:11 dist-test-slave-42z9 krb5kdc[13098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.100.2@KRBTEST.COM for oryx/127.31.100.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 05:21:11 dist-test-slave-42z9 krb5kdc[13098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.100.2@KRBTEST.COM for oryx/127.31.100.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 05:21:11.685796 14881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:21:11.677870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.62:41525 (local address 127.31.100.2:52291)
0812 05:21:11.678009 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0812 05:21:11.678021 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0812 05:21:11.678091 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:21:11.678235 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:21:11.678236 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:21:11.678406 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0812 05:21:11.678460 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:21:11.678462 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:21:11.679081 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:21:11.679082 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:21:11.679625 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:21:11.679627 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:21:11.680066 (+ 439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:21:11.680067 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:21:11.680130 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:21:11.683982 (+ 3852us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 05:21:11.683987 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0812 05:21:11.685736 (+ 1749us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.100.62:41525: Server oryx/127.31.100.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":73,"threads_started":1}
W20260812 05:21:11.685912 14880 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 05:21:11.677921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.100.2:52291 (local address 127.31.100.62:41525)
0812 05:21:11.677983 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0812 05:21:11.677985 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 05:21:11.678090 (+ 105us) server_negotiation.cc:407] Connection header received
0812 05:21:11.678155 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:21:11.678156 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:21:11.678170 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:21:11.678218 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:21:11.678521 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:21:11.679024 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:21:11.679704 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:21:11.679905 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:21:11.679960 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:21:11.685842 (+ 5882us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.100.2:52291: BlockingRecv error: recv got EOF from 127.31.100.2:52291 (error 108)
Metrics: {"server-negotiator.queue_time_us":28}
Aug 12 05:21:11 dist-test-slave-42z9 krb5kdc[13098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.100.3@KRBTEST.COM for oryx/127.31.100.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 05:21:11 dist-test-slave-42z9 krb5kdc[13098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.100.3@KRBTEST.COM for oryx/127.31.100.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 05:21:11.799068 14882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:21:11.794869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.62:41525 (local address 127.31.100.3:57639)
0812 05:21:11.795004 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0812 05:21:11.795015 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0812 05:21:11.795073 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:21:11.795183 (+ 110us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:21:11.795184 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:21:11.795356 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0812 05:21:11.795407 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:21:11.795409 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:21:11.796013 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:21:11.796014 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:21:11.796546 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:21:11.796547 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:21:11.796833 (+ 286us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:21:11.796834 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:21:11.796871 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:21:11.797274 (+ 403us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 05:21:11.797278 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0812 05:21:11.798992 (+ 1714us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.100.62:41525: Server oryx/127.31.100.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":70,"threads_started":1}
W20260812 05:21:11.799137 14880 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 05:21:11.794905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.100.3:57639 (local address 127.31.100.62:41525)
0812 05:21:11.794955 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0812 05:21:11.794957 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 05:21:11.795067 (+ 110us) server_negotiation.cc:407] Connection header received
0812 05:21:11.795122 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:21:11.795122 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:21:11.795134 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:21:11.795169 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:21:11.795465 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:21:11.795958 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:21:11.796604 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:21:11.796782 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:21:11.796829 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:21:11.799084 (+ 2255us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.100.3:57639: BlockingRecv error: recv got EOF from 127.31.100.3:57639 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
Aug 12 05:21:12 dist-test-slave-42z9 krb5kdc[13098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.100.1@KRBTEST.COM for oryx/127.31.100.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 05:21:12 dist-test-slave-42z9 krb5kdc[13098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.100.1@KRBTEST.COM for oryx/127.31.100.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 05:21:12.546944 14883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:21:12.542674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.62:41525 (local address 127.31.100.1:39807)
0812 05:21:12.542815 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0812 05:21:12.542826 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0812 05:21:12.542874 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:21:12.543003 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:21:12.543004 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:21:12.543165 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0812 05:21:12.543214 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:21:12.543216 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:21:12.543831 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:21:12.543831 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:21:12.544361 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:21:12.544362 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:21:12.544667 (+ 305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:21:12.544668 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:21:12.544704 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:21:12.545096 (+ 392us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 05:21:12.545100 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0812 05:21:12.546865 (+ 1765us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.100.62:41525: Server oryx/127.31.100.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":65,"threads_started":1}
W20260812 05:21:12.547026 14884 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 05:21:12.542732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.100.1:39807 (local address 127.31.100.62:41525)
0812 05:21:12.542847 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0812 05:21:12.542849 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 05:21:12.542866 (+ 17us) server_negotiation.cc:407] Connection header received
0812 05:21:12.542932 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:21:12.542932 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:21:12.542943 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:21:12.542990 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:21:12.543265 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:21:12.543775 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:21:12.544424 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:21:12.544615 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:21:12.544664 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:21:12.546962 (+ 2298us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.100.1:39807: BlockingRecv error: recv got EOF from 127.31.100.1:39807 (error 108)
Metrics: {"server-negotiator.queue_time_us":95,"thread_start_us":56,"threads_started":1}
Aug 12 05:21:12 dist-test-slave-42z9 krb5kdc[13098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.100.2@KRBTEST.COM for oryx/127.31.100.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 05:21:12 dist-test-slave-42z9 krb5kdc[13098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.100.2@KRBTEST.COM for oryx/127.31.100.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 05:21:12.690801 14888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:21:12.686321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.62:41525 (local address 127.31.100.2:55899)
0812 05:21:12.686516 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0812 05:21:12.686526 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0812 05:21:12.686589 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:21:12.686711 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:21:12.686711 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:21:12.686878 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0812 05:21:12.686929 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:21:12.686931 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:21:12.687653 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:21:12.687654 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:21:12.688198 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:21:12.688201 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:21:12.688549 (+ 348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:21:12.688550 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:21:12.688589 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:21:12.689007 (+ 418us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 05:21:12.689012 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0812 05:21:12.690724 (+ 1712us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.100.62:41525: Server oryx/127.31.100.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":74,"threads_started":1}
W20260812 05:21:12.690881 14884 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 05:21:12.686424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.100.2:55899 (local address 127.31.100.62:41525)
0812 05:21:12.686479 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0812 05:21:12.686481 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 05:21:12.686571 (+ 90us) server_negotiation.cc:407] Connection header received
0812 05:21:12.686654 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:21:12.686655 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:21:12.686666 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:21:12.686700 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:21:12.687022 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:21:12.687572 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:21:12.688274 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:21:12.688490 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:21:12.688545 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:21:12.690828 (+ 2283us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.100.2:55899: BlockingRecv error: recv got EOF from 127.31.100.2:55899 (error 108)
Metrics: {"server-negotiator.queue_time_us":28}
I20260812 05:21:12.723242 32144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0waFvT/build/release/bin/kudu with pid 14110
I20260812 05:21:12.743602 32144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0waFvT/build/release/bin/kudu with pid 14354
I20260812 05:21:12.751259 32144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0waFvT/build/release/bin/kudu with pid 14568
I20260812 05:21:12.760739 32144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0waFvT/build/release/bin/kudu with pid 13968
2026-08-12T05:21:12Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (17545 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-08-12T05:21:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T05:21:12Z Disabled control of system clock
I20260812 05:21:12.792658 32144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0waFvT/build/release/bin/kudu
/tmp/dist-test-task0waFvT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786512017613366-32144-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786512017613366-32144-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786512017613366-32144-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786512017613366-32144-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.100.62:40685
--webserver_interface=127.31.100.62
--webserver_port=0
--builtin_ntp_servers=127.31.100.20:35075
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.100.62:40685
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260812 05:21:12.873821 14901 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 05:21:12.874030 14901 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 05:21:12.874058 14901 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 05:21:12.875206 14901 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 05:21:12.875249 14901 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 05:21:12.875267 14901 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 05:21:12.875281 14901 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 05:21:12.875295 14901 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 05:21:12.876617 14901 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.100.20:35075
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786512017613366-32144-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786512017613366-32144-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.100.62:40685
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.100.62:40685
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786512017613366-32144-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.100.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14901
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786512017613366-32144-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 04:20:34 UTC on ccycloud.quasar-phqdqp.root.comops.site
build id 4581871
I20260812 05:21:12.876871 14901 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 05:21:12.877132 14901 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 05:21:12.882328 14910 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 05:21:12.882392 14915 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 05:21:12.882345 14913 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 05:21:12.882400 14901 server_base.cc:1034] running on GCE node
I20260812 05:21:12.882766 14901 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 05:21:12.883047 14901 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 05:21:12.884209 14901 hybrid_clock.cc:648] HybridClock initialized: now 1786512072884193 us; error 30 us; skew 500 ppm
I20260812 05:21:12.885612 14901 webserver.cc:460] Webserver started at http://127.31.100.62:46457/ using document root <none> and password file <none>
I20260812 05:21:12.885840 14901 fs_manager.cc:359] Metadata directory not provided
I20260812 05:21:12.885893 14901 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 05:21:12.886021 14901 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 05:21:12.887074 14901 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786512017613366-32144-0/minicluster-data/master-0/data/instance:
uuid: "69dcb9fcd80149aca315347fb5245287"
format_stamp: "Formatted at 2026-08-12 05:21:12 on dist-test-slave-42z9"
server_key: "b9d6457c00c8bbc19beabed5c2293ff6"
server_key_iv: "6b5bbd29477c9c22d92be83f86875376"
server_key_version: "encryptionkey@0"
I20260812 05:21:12.887414 14901 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786512017613366-32144-0/minicluster-data/master-0/wal/instance:
uuid: "69dcb9fcd80149aca315347fb5245287"
format_stamp: "Formatted at 2026-08-12 05:21:12 on dist-test-slave-42z9"
server_key: "b9d6457c00c8bbc19beabed5c2293ff6"
server_key_iv: "6b5bbd29477c9c22d92be83f86875376"
server_key_version: "encryptionkey@0"
I20260812 05:21:12.888676 14901 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260812 05:21:12.889421 14922 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 05:21:12.889639 14901 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260812 05:21:12.889734 14901 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786512017613366-32144-0/minicluster-data/master-0/data,/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786512017613366-32144-0/minicluster-data/master-0/wal
uuid: "69dcb9fcd80149aca315347fb5245287"
format_stamp: "Formatted at 2026-08-12 05:21:12 on dist-test-slave-42z9"
server_key: "b9d6457c00c8bbc19beabed5c2293ff6"
server_key_iv: "6b5bbd29477c9c22d92be83f86875376"
server_key_version: "encryptionkey@0"
I20260812 05:21:12.889796 14901 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786512017613366-32144-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786512017613366-32144-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786512017613366-32144-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 05:21:12.916246 14901 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 05:21:12.916594 14901 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 05:21:12.916730 14901 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 05:21:12.920750 14901 rpc_server.cc:307] RPC server started. Bound to: 127.31.100.62:40685
I20260812 05:21:12.920792 14997 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.100.62:40685 every 8 connection(s)
I20260812 05:21:12.921139 14901 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786512017613366-32144-0/minicluster-data/master-0/data/info.pb
I20260812 05:21:12.921838 15003 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 05:21:12.924396 15003 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 69dcb9fcd80149aca315347fb5245287: Bootstrap starting.
I20260812 05:21:12.924963 15003 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 69dcb9fcd80149aca315347fb5245287: Neither blocks nor log segments found. Creating new log.
I20260812 05:21:12.925208 15003 log.cc:824] T 00000000000000000000000000000000 P 69dcb9fcd80149aca315347fb5245287: Log is configured to *not* fsync() on all Append() calls
I20260812 05:21:12.925918 15003 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 69dcb9fcd80149aca315347fb5245287: No bootstrap required, opened a new log
I20260812 05:21:12.927142 32144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0waFvT/build/release/bin/kudu as pid 14901
I20260812 05:21:12.927237 32144 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786512017613366-32144-0/minicluster-data/master-0/wal/instance
I20260812 05:21:12.927443 32144 external_mini_cluster.cc:1428] Setting key 93fc6f562ae291ebb1c094ffe80315dc
I20260812 05:21:12.927695 15003 raft_consensus.cc:348] T 00000000000000000000000000000000 P 69dcb9fcd80149aca315347fb5245287 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69dcb9fcd80149aca315347fb5245287" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 40685 } }
I20260812 05:21:12.927842 15003 raft_consensus.cc:374] T 00000000000000000000000000000000 P 69dcb9fcd80149aca315347fb5245287 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 05:21:12.927887 15003 raft_consensus.cc:729] T 00000000000000000000000000000000 P 69dcb9fcd80149aca315347fb5245287 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69dcb9fcd80149aca315347fb5245287, State: Initialized, Role: FOLLOWER
I20260812 05:21:12.928030 15003 consensus_queue.cc:260] T 00000000000000000000000000000000 P 69dcb9fcd80149aca315347fb5245287 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69dcb9fcd80149aca315347fb5245287" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 40685 } }
I20260812 05:21:12.928203 15003 raft_consensus.cc:388] T 00000000000000000000000000000000 P 69dcb9fcd80149aca315347fb5245287 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 05:21:12.928248 15003 raft_consensus.cc:482] T 00000000000000000000000000000000 P 69dcb9fcd80149aca315347fb5245287 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 05:21:12.928292 15003 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 69dcb9fcd80149aca315347fb5245287 [term 0 FOLLOWER]: Advancing to term 1
I20260812 05:21:12.928925 15003 raft_consensus.cc:504] T 00000000000000000000000000000000 P 69dcb9fcd80149aca315347fb5245287 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69dcb9fcd80149aca315347fb5245287" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 40685 } }
I20260812 05:21:12.929034 15003 leader_election.cc:302] T 00000000000000000000000000000000 P 69dcb9fcd80149aca315347fb5245287 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 69dcb9fcd80149aca315347fb5245287; no voters:
I20260812 05:21:12.929183 15003 leader_election.cc:288] T 00000000000000000000000000000000 P 69dcb9fcd80149aca315347fb5245287 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 05:21:12.929239 15011 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 69dcb9fcd80149aca315347fb5245287 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 05:21:12.929392 15011 raft_consensus.cc:686] T 00000000000000000000000000000000 P 69dcb9fcd80149aca315347fb5245287 [term 1 LEADER]: Becoming Leader. State: Replica: 69dcb9fcd80149aca315347fb5245287, State: Running, Role: LEADER
I20260812 05:21:12.929471 15003 sys_catalog.cc:564] T 00000000000000000000000000000000 P 69dcb9fcd80149aca315347fb5245287 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 05:21:12.929561 15011 consensus_queue.cc:237] T 00000000000000000000000000000000 P 69dcb9fcd80149aca315347fb5245287 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69dcb9fcd80149aca315347fb5245287" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 40685 } }
I20260812 05:21:12.929909 15011 sys_catalog.cc:455] T 00000000000000000000000000000000 P 69dcb9fcd80149aca315347fb5245287 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 69dcb9fcd80149aca315347fb5245287. Latest consensus state: current_term: 1 leader_uuid: "69dcb9fcd80149aca315347fb5245287" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69dcb9fcd80149aca315347fb5245287" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 40685 } } }
I20260812 05:21:12.929981 15012 sys_catalog.cc:455] T 00000000000000000000000000000000 P 69dcb9fcd80149aca315347fb5245287 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "69dcb9fcd80149aca315347fb5245287" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69dcb9fcd80149aca315347fb5245287" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 40685 } } }
I20260812 05:21:12.930059 15011 sys_catalog.cc:458] T 00000000000000000000000000000000 P 69dcb9fcd80149aca315347fb5245287 [sys.catalog]: This master's current role is: LEADER
I20260812 05:21:12.930150 15012 sys_catalog.cc:458] T 00000000000000000000000000000000 P 69dcb9fcd80149aca315347fb5245287 [sys.catalog]: This master's current role is: LEADER
I20260812 05:21:12.930354 15030 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 05:21:12.931166 15030 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 05:21:12.933753 15009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:21:12.927960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49572 (local address 127.31.100.62:40685)
0812 05:21:12.928206 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0812 05:21:12.928216 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 05:21:12.928434 (+ 218us) server_negotiation.cc:407] Connection header received
0812 05:21:12.928565 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:21:12.928572 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:21:12.928661 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:21:12.928818 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:21:12.929074 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:21:12.930712 (+ 1638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:21:12.931781 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:21:12.932331 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:21:12.932465 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:21:12.933099 (+ 634us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 05:21:12.933102 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 05:21:12.933102 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 05:21:12.933272 (+ 170us) server_negotiation.cc:922] calling sasl_server_start()
0812 05:21:12.933304 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 05:21:12.933314 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 05:21:12.933363 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0812 05:21:12.933478 (+ 115us) server_negotiation.cc:299] Negotiation successful
0812 05:21:12.933549 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":92,"threads_started":1}
I20260812 05:21:12.937845 15030 catalog_manager.cc:1347] Generated new cluster ID: c6de6068b81d49bf95e15826abf7622d
I20260812 05:21:12.937896 15030 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 05:21:12.961280 15030 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 05:21:12.961826 15030 catalog_manager.cc:1504] Loading token signing keys...
I20260812 05:21:12.963797 15030 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 69dcb9fcd80149aca315347fb5245287: Generated new TSK 0
I20260812 05:21:12.963961 15030 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 05:21:12.967633 32144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0waFvT/build/release/bin/kudu
/tmp/dist-test-task0waFvT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786512017613366-32144-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786512017613366-32144-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786512017613366-32144-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786512017613366-32144-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.100.1:0
--local_ip_for_outbound_sockets=127.31.100.1
--webserver_interface=127.31.100.1
--webserver_port=0
--tserver_master_addrs=127.31.100.62:40685
--builtin_ntp_servers=127.31.100.20:35075
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260812 05:21:13.068377 15053 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 05:21:13.068593 15053 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 05:21:13.068622 15053 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 05:21:13.069780 15053 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 05:21:13.069819 15053 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 05:21:13.069851 15053 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.100.1
I20260812 05:21:13.071162 15053 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.100.20:35075
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786512017613366-32144-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786512017613366-32144-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.100.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786512017613366-32144-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.100.1
--webserver_port=0
--tserver_master_addrs=127.31.100.62:40685
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15053
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.100.1
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786512017613366-32144-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 04:20:34 UTC on ccycloud.quasar-phqdqp.root.comops.site
build id 4581871
I20260812 05:21:13.071401 15053 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 05:21:13.071691 15053 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 05:21:13.076522 15062 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 05:21:13.076555 15064 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 05:21:13.076520 15066 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 05:21:13.076779 15053 server_base.cc:1034] running on GCE node
I20260812 05:21:13.076978 15053 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 05:21:13.077226 15053 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 05:21:13.078364 15053 hybrid_clock.cc:648] HybridClock initialized: now 1786512073078352 us; error 28 us; skew 500 ppm
I20260812 05:21:13.079587 15053 webserver.cc:460] Webserver started at http://127.31.100.1:35973/ using document root <none> and password file <none>
I20260812 05:21:13.079793 15053 fs_manager.cc:359] Metadata directory not provided
I20260812 05:21:13.079833 15053 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 05:21:13.079923 15053 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 05:21:13.080902 15053 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786512017613366-32144-0/minicluster-data/ts-0/data/instance:
uuid: "f44e8ab6c6e442bf88e3d534df274d02"
format_stamp: "Formatted at 2026-08-12 05:21:13 on dist-test-slave-42z9"
server_key: "589a16966cfae170f4b7b44f7fa103f0"
server_key_iv: "2b20319d0a106ad9ca951bad8378c490"
server_key_version: "encryptionkey@0"
I20260812 05:21:13.081188 15053 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786512017613366-32144-0/minicluster-data/ts-0/wal/instance:
uuid: "f44e8ab6c6e442bf88e3d534df274d02"
format_stamp: "Formatted at 2026-08-12 05:21:13 on dist-test-slave-42z9"
server_key: "589a16966cfae170f4b7b44f7fa103f0"
server_key_iv: "2b20319d0a106ad9ca951bad8378c490"
server_key_version: "encryptionkey@0"
I20260812 05:21:13.082396 15053 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260812 05:21:13.083061 15074 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 05:21:13.083212 15053 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260812 05:21:13.083295 15053 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786512017613366-32144-0/minicluster-data/ts-0/data,/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786512017613366-32144-0/minicluster-data/ts-0/wal
uuid: "f44e8ab6c6e442bf88e3d534df274d02"
format_stamp: "Formatted at 2026-08-12 05:21:13 on dist-test-slave-42z9"
server_key: "589a16966cfae170f4b7b44f7fa103f0"
server_key_iv: "2b20319d0a106ad9ca951bad8378c490"
server_key_version: "encryptionkey@0"
I20260812 05:21:13.083360 15053 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786512017613366-32144-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786512017613366-32144-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786512017613366-32144-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 05:21:13.095389 15053 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 05:21:13.095679 15053 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 05:21:13.095796 15053 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 05:21:13.095991 15053 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 05:21:13.096289 15053 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 05:21:13.096323 15053 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 05:21:13.096349 15053 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 05:21:13.096375 15053 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 05:21:13.101897 15053 rpc_server.cc:307] RPC server started. Bound to: 127.31.100.1:35553
I20260812 05:21:13.101961 15254 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.100.1:35553 every 8 connection(s)
I20260812 05:21:13.102258 15053 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786512017613366-32144-0/minicluster-data/ts-0/data/info.pb
I20260812 05:21:13.103978 32144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0waFvT/build/release/bin/kudu as pid 15053
I20260812 05:21:13.104060 32144 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786512017613366-32144-0/minicluster-data/ts-0/wal/instance
I20260812 05:21:13.104233 32144 external_mini_cluster.cc:1428] Setting key 72b03cbc46d0cb5ade9d9e65558b29da
I20260812 05:21:13.105338 32144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0waFvT/build/release/bin/kudu
/tmp/dist-test-task0waFvT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786512017613366-32144-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786512017613366-32144-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786512017613366-32144-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786512017613366-32144-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.100.2:0
--local_ip_for_outbound_sockets=127.31.100.2
--webserver_interface=127.31.100.2
--webserver_port=0
--tserver_master_addrs=127.31.100.62:40685
--builtin_ntp_servers=127.31.100.20:35075
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260812 05:21:13.107681 15009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:21:13.102898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.100.1:57537 (local address 127.31.100.62:40685)
0812 05:21:13.102976 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0812 05:21:13.102979 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 05:21:13.103263 (+ 284us) server_negotiation.cc:407] Connection header received
0812 05:21:13.103761 (+ 498us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:21:13.103762 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:21:13.103777 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:21:13.103849 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:21:13.104611 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:21:13.105264 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:21:13.106333 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:21:13.106602 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:21:13.106668 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:21:13.107428 (+ 760us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 05:21:13.107429 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 05:21:13.107429 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 05:21:13.107435 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0812 05:21:13.107452 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 05:21:13.107457 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 05:21:13.107512 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0812 05:21:13.107568 (+ 56us) server_negotiation.cc:299] Negotiation successful
0812 05:21:13.107601 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260812 05:21:13.107810 15257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:21:13.102896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.62:40685 (local address 127.31.100.1:57537)
0812 05:21:13.103114 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0812 05:21:13.103186 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0812 05:21:13.103675 (+ 489us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:21:13.103903 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:21:13.103906 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:21:13.103915 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0812 05:21:13.104547 (+ 632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:21:13.104554 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:21:13.105367 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:21:13.105368 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:21:13.106242 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:21:13.106245 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:21:13.106683 (+ 438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:21:13.106684 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:21:13.106791 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:21:13.107296 (+ 505us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 05:21:13.107296 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0812 05:21:13.107325 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 05:21:13.107348 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 05:21:13.107512 (+ 164us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 05:21:13.107514 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 05:21:13.107517 (+ 3us) client_negotiation.cc:769] Sending connection context
0812 05:21:13.107559 (+ 42us) client_negotiation.cc:240] Negotiation successful
0812 05:21:13.107635 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":91,"threads_started":1}
I20260812 05:21:13.108265 15255 heartbeater.cc:347] Connected to a master server at 127.31.100.62:40685
I20260812 05:21:13.108363 15255 heartbeater.cc:464] Registering TS with master...
I20260812 05:21:13.108552 15255 heartbeater.cc:511] Master 127.31.100.62:40685 requested a full tablet report, sending...
I20260812 05:21:13.109081 14952 ts_manager.cc:194] Registered new tserver with Master: f44e8ab6c6e442bf88e3d534df274d02 (127.31.100.1:35553)
I20260812 05:21:13.110018 14952 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.31.100.1:57537
W20260812 05:21:13.190807 15258 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 05:21:13.191004 15258 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 05:21:13.191025 15258 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 05:21:13.192137 15258 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 05:21:13.192175 15258 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 05:21:13.192198 15258 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.100.2
I20260812 05:21:13.193552 15258 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.100.20:35075
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786512017613366-32144-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786512017613366-32144-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.100.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786512017613366-32144-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.100.2
--webserver_port=0
--tserver_master_addrs=127.31.100.62:40685
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15258
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.100.2
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786512017613366-32144-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 04:20:34 UTC on ccycloud.quasar-phqdqp.root.comops.site
build id 4581871
I20260812 05:21:13.193754 15258 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 05:21:13.194001 15258 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 05:21:13.198800 15270 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 05:21:13.198913 15258 server_base.cc:1034] running on GCE node
W20260812 05:21:13.198812 15271 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 05:21:13.198800 15273 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 05:21:13.199213 15258 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 05:21:13.199461 15258 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 05:21:13.200588 15258 hybrid_clock.cc:648] HybridClock initialized: now 1786512073200581 us; error 26 us; skew 500 ppm
I20260812 05:21:13.201880 15258 webserver.cc:460] Webserver started at http://127.31.100.2:41531/ using document root <none> and password file <none>
I20260812 05:21:13.202111 15258 fs_manager.cc:359] Metadata directory not provided
I20260812 05:21:13.202160 15258 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 05:21:13.202278 15258 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 05:21:13.203315 15258 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786512017613366-32144-0/minicluster-data/ts-1/data/instance:
uuid: "3d3137a25b78430b84980d489139793f"
format_stamp: "Formatted at 2026-08-12 05:21:13 on dist-test-slave-42z9"
server_key: "5a01e0d607a1fdae740a1280f46e2711"
server_key_iv: "bfa530aeb733c32b8bb7c7211db0312b"
server_key_version: "encryptionkey@0"
I20260812 05:21:13.203683 15258 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786512017613366-32144-0/minicluster-data/ts-1/wal/instance:
uuid: "3d3137a25b78430b84980d489139793f"
format_stamp: "Formatted at 2026-08-12 05:21:13 on dist-test-slave-42z9"
server_key: "5a01e0d607a1fdae740a1280f46e2711"
server_key_iv: "bfa530aeb733c32b8bb7c7211db0312b"
server_key_version: "encryptionkey@0"
I20260812 05:21:13.204859 15258 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260812 05:21:13.205597 15283 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 05:21:13.205734 15258 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260812 05:21:13.205812 15258 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786512017613366-32144-0/minicluster-data/ts-1/data,/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786512017613366-32144-0/minicluster-data/ts-1/wal
uuid: "3d3137a25b78430b84980d489139793f"
format_stamp: "Formatted at 2026-08-12 05:21:13 on dist-test-slave-42z9"
server_key: "5a01e0d607a1fdae740a1280f46e2711"
server_key_iv: "bfa530aeb733c32b8bb7c7211db0312b"
server_key_version: "encryptionkey@0"
I20260812 05:21:13.205878 15258 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786512017613366-32144-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786512017613366-32144-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786512017613366-32144-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 05:21:13.215631 15258 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 05:21:13.215857 15258 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 05:21:13.215974 15258 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 05:21:13.216157 15258 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 05:21:13.216440 15258 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 05:21:13.216475 15258 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 05:21:13.216500 15258 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 05:21:13.216517 15258 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 05:21:13.221911 15258 rpc_server.cc:307] RPC server started. Bound to: 127.31.100.2:44345
I20260812 05:21:13.221966 15473 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.100.2:44345 every 8 connection(s)
I20260812 05:21:13.222235 15258 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786512017613366-32144-0/minicluster-data/ts-1/data/info.pb
I20260812 05:21:13.227216 15009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:21:13.222927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.100.2:36675 (local address 127.31.100.62:40685)
0812 05:21:13.222989 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0812 05:21:13.222991 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 05:21:13.223194 (+ 203us) server_negotiation.cc:407] Connection header received
0812 05:21:13.223739 (+ 545us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:21:13.223741 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:21:13.223753 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:21:13.223806 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:21:13.224472 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:21:13.224973 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:21:13.225972 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:21:13.226168 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:21:13.226224 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:21:13.226955 (+ 731us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 05:21:13.226963 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 05:21:13.226963 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 05:21:13.226968 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0812 05:21:13.226984 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 05:21:13.226988 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 05:21:13.227031 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0812 05:21:13.227104 (+ 73us) server_negotiation.cc:299] Negotiation successful
0812 05:21:13.227136 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260812 05:21:13.227393 15477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:21:13.222885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.62:40685 (local address 127.31.100.2:36675)
0812 05:21:13.223082 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0812 05:21:13.223112 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0812 05:21:13.223630 (+ 518us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:21:13.223880 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:21:13.223884 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:21:13.223894 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0812 05:21:13.224382 (+ 488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:21:13.224386 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:21:13.225055 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:21:13.225056 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:21:13.225887 (+ 831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:21:13.225889 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:21:13.226240 (+ 351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:21:13.226241 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:21:13.226360 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:21:13.226832 (+ 472us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 05:21:13.226832 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0812 05:21:13.226859 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 05:21:13.226880 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 05:21:13.227038 (+ 158us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 05:21:13.227043 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 05:21:13.227046 (+ 3us) client_negotiation.cc:769] Sending connection context
0812 05:21:13.227103 (+ 57us) client_negotiation.cc:240] Negotiation successful
0812 05:21:13.227203 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":87,"threads_started":1}
I20260812 05:21:13.227795 15475 heartbeater.cc:347] Connected to a master server at 127.31.100.62:40685
I20260812 05:21:13.227876 15475 heartbeater.cc:464] Registering TS with master...
I20260812 05:21:13.228022 15475 heartbeater.cc:511] Master 127.31.100.62:40685 requested a full tablet report, sending...
I20260812 05:21:13.228317 14952 ts_manager.cc:194] Registered new tserver with Master: 3d3137a25b78430b84980d489139793f (127.31.100.2:44345)
I20260812 05:21:13.228642 14952 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.31.100.2:36675
I20260812 05:21:13.231320 32144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0waFvT/build/release/bin/kudu as pid 15258
I20260812 05:21:13.231444 32144 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786512017613366-32144-0/minicluster-data/ts-1/wal/instance
I20260812 05:21:13.231637 32144 external_mini_cluster.cc:1428] Setting key 702bcafc2d8bd7845e2038aade440d3b
I20260812 05:21:13.232622 32144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0waFvT/build/release/bin/kudu
/tmp/dist-test-task0waFvT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786512017613366-32144-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786512017613366-32144-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786512017613366-32144-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786512017613366-32144-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.100.3:0
--local_ip_for_outbound_sockets=127.31.100.3
--webserver_interface=127.31.100.3
--webserver_port=0
--tserver_master_addrs=127.31.100.62:40685
--builtin_ntp_servers=127.31.100.20:35075
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260812 05:21:13.312741 15478 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 05:21:13.312930 15478 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 05:21:13.312949 15478 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 05:21:13.314085 15478 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 05:21:13.314126 15478 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 05:21:13.314148 15478 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.100.3
I20260812 05:21:13.315408 15478 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.100.20:35075
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786512017613366-32144-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786512017613366-32144-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.100.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786512017613366-32144-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.100.3
--webserver_port=0
--tserver_master_addrs=127.31.100.62:40685
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15478
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.100.3
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786512017613366-32144-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 04:20:34 UTC on ccycloud.quasar-phqdqp.root.comops.site
build id 4581871
I20260812 05:21:13.315639 15478 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 05:21:13.315905 15478 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 05:21:13.320629 15494 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 05:21:13.320664 15492 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 05:21:13.320781 15478 server_base.cc:1034] running on GCE node
W20260812 05:21:13.320853 15491 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 05:21:13.321053 15478 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 05:21:13.321265 15478 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 05:21:13.322429 15478 hybrid_clock.cc:648] HybridClock initialized: now 1786512073322393 us; error 24 us; skew 500 ppm
I20260812 05:21:13.323634 15478 webserver.cc:460] Webserver started at http://127.31.100.3:42953/ using document root <none> and password file <none>
I20260812 05:21:13.323843 15478 fs_manager.cc:359] Metadata directory not provided
I20260812 05:21:13.323889 15478 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 05:21:13.324016 15478 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 05:21:13.325011 15478 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786512017613366-32144-0/minicluster-data/ts-2/data/instance:
uuid: "2baed3ee5c484ac48ae8b90abd401009"
format_stamp: "Formatted at 2026-08-12 05:21:13 on dist-test-slave-42z9"
server_key: "c6a4f898bf48566126014419620a567f"
server_key_iv: "4fcbbc1c920af9e32a9e5daaf092c4c1"
server_key_version: "encryptionkey@0"
I20260812 05:21:13.325357 15478 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786512017613366-32144-0/minicluster-data/ts-2/wal/instance:
uuid: "2baed3ee5c484ac48ae8b90abd401009"
format_stamp: "Formatted at 2026-08-12 05:21:13 on dist-test-slave-42z9"
server_key: "c6a4f898bf48566126014419620a567f"
server_key_iv: "4fcbbc1c920af9e32a9e5daaf092c4c1"
server_key_version: "encryptionkey@0"
I20260812 05:21:13.326645 15478 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260812 05:21:13.327273 15502 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 05:21:13.327428 15478 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260812 05:21:13.327512 15478 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786512017613366-32144-0/minicluster-data/ts-2/data,/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786512017613366-32144-0/minicluster-data/ts-2/wal
uuid: "2baed3ee5c484ac48ae8b90abd401009"
format_stamp: "Formatted at 2026-08-12 05:21:13 on dist-test-slave-42z9"
server_key: "c6a4f898bf48566126014419620a567f"
server_key_iv: "4fcbbc1c920af9e32a9e5daaf092c4c1"
server_key_version: "encryptionkey@0"
I20260812 05:21:13.327577 15478 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786512017613366-32144-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786512017613366-32144-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786512017613366-32144-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 05:21:13.344630 15478 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 05:21:13.344895 15478 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 05:21:13.345018 15478 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 05:21:13.345222 15478 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 05:21:13.345618 15478 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 05:21:13.345654 15478 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 05:21:13.345681 15478 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 05:21:13.345698 15478 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 05:21:13.351094 15478 rpc_server.cc:307] RPC server started. Bound to: 127.31.100.3:40577
I20260812 05:21:13.351147 15700 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.100.3:40577 every 8 connection(s)
I20260812 05:21:13.351492 15478 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786512017613366-32144-0/minicluster-data/ts-2/data/info.pb
I20260812 05:21:13.356343 15009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:21:13.352145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.100.3:60199 (local address 127.31.100.62:40685)
0812 05:21:13.352203 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0812 05:21:13.352206 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 05:21:13.352441 (+ 235us) server_negotiation.cc:407] Connection header received
0812 05:21:13.352892 (+ 451us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:21:13.352901 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:21:13.352911 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:21:13.352960 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:21:13.353666 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:21:13.354172 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:21:13.355142 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:21:13.355323 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:21:13.355387 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:21:13.356127 (+ 740us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 05:21:13.356128 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 05:21:13.356128 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 05:21:13.356133 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0812 05:21:13.356145 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 05:21:13.356149 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 05:21:13.356191 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0812 05:21:13.356256 (+ 65us) server_negotiation.cc:299] Negotiation successful
0812 05:21:13.356282 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260812 05:21:13.356510 15703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:21:13.352135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.62:40685 (local address 127.31.100.3:60199)
0812 05:21:13.352324 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0812 05:21:13.352357 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0812 05:21:13.352814 (+ 457us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:21:13.353014 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:21:13.353017 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:21:13.353025 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0812 05:21:13.353556 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:21:13.353561 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:21:13.354280 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:21:13.354282 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:21:13.355069 (+ 787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:21:13.355071 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:21:13.355386 (+ 315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:21:13.355387 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:21:13.355499 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:21:13.356004 (+ 505us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 05:21:13.356004 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0812 05:21:13.356032 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 05:21:13.356056 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 05:21:13.356198 (+ 142us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 05:21:13.356201 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 05:21:13.356203 (+ 2us) client_negotiation.cc:769] Sending connection context
0812 05:21:13.356250 (+ 47us) client_negotiation.cc:240] Negotiation successful
0812 05:21:13.356327 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":90,"threads_started":1}
I20260812 05:21:13.356922 15701 heartbeater.cc:347] Connected to a master server at 127.31.100.62:40685
I20260812 05:21:13.357012 15701 heartbeater.cc:464] Registering TS with master...
I20260812 05:21:13.357180 15701 heartbeater.cc:511] Master 127.31.100.62:40685 requested a full tablet report, sending...
I20260812 05:21:13.357534 14952 ts_manager.cc:194] Registered new tserver with Master: 2baed3ee5c484ac48ae8b90abd401009 (127.31.100.3:40577)
I20260812 05:21:13.357863 14952 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.31.100.3:60199
I20260812 05:21:13.357872 32144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0waFvT/build/release/bin/kudu as pid 15478
I20260812 05:21:13.357936 32144 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786512017613366-32144-0/minicluster-data/ts-2/wal/instance
I20260812 05:21:13.358167 32144 external_mini_cluster.cc:1428] Setting key ec8ed2b295627c4b0c2b6e3348207c55
I20260812 05:21:13.358934 32144 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260812 05:21:13.360563 15009 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 05:21:13.360109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49584 (local address 127.31.100.62:40685)
0812 05:21:13.360156 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0812 05:21:13.360158 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 05:21:13.360274 (+ 116us) server_negotiation.cc:407] Connection header received
0812 05:21:13.360325 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:21:13.360325 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:21:13.360337 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:21:13.360381 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:21:13.360505 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49584: BlockingRecv error: recv got EOF from 127.0.0.1:49584 (error 108)
Metrics: {"server-negotiator.queue_time_us":26}
I20260812 05:21:13.360615 32144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0waFvT/build/release/bin/kudu with pid 15053
I20260812 05:21:13.381947 32144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0waFvT/build/release/bin/kudu with pid 15258
I20260812 05:21:13.390223 32144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0waFvT/build/release/bin/kudu with pid 15478
I20260812 05:21:13.399753 32144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0waFvT/build/release/bin/kudu with pid 14901
2026-08-12T05:21:13Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (636 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-08-12T05:21:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T05:21:13Z Disabled control of system clock
I20260812 05:21:13.428947 32144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0waFvT/build/release/bin/kudu
/tmp/dist-test-task0waFvT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786512017613366-32144-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786512017613366-32144-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786512017613366-32144-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786512017613366-32144-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.100.62:37181
--webserver_interface=127.31.100.62
--webserver_port=0
--builtin_ntp_servers=127.31.100.20:35499
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.100.62:37181
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260812 05:21:13.509603 15721 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 05:21:13.509796 15721 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 05:21:13.509815 15721 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 05:21:13.510900 15721 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 05:21:13.510941 15721 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 05:21:13.510952 15721 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 05:21:13.510963 15721 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 05:21:13.510974 15721 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 05:21:13.512245 15721 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.100.20:35499
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786512017613366-32144-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786512017613366-32144-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.100.62:37181
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.100.62:37181
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786512017613366-32144-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.100.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15721
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786512017613366-32144-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 04:20:34 UTC on ccycloud.quasar-phqdqp.root.comops.site
build id 4581871
I20260812 05:21:13.512429 15721 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 05:21:13.512691 15721 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 05:21:13.517292 15730 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 05:21:13.517386 15731 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 05:21:13.517321 15736 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 05:21:13.517546 15721 server_base.cc:1034] running on GCE node
I20260812 05:21:13.517767 15721 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 05:21:13.518019 15721 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 05:21:13.519156 15721 hybrid_clock.cc:648] HybridClock initialized: now 1786512073519143 us; error 30 us; skew 500 ppm
I20260812 05:21:13.520406 15721 webserver.cc:460] Webserver started at http://127.31.100.62:37777/ using document root <none> and password file <none>
I20260812 05:21:13.520619 15721 fs_manager.cc:359] Metadata directory not provided
I20260812 05:21:13.520666 15721 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 05:21:13.520809 15721 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 05:21:13.521862 15721 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786512017613366-32144-0/minicluster-data/master-0/data/instance:
uuid: "9aa2b418f50b4b828e756d747ee2cec3"
format_stamp: "Formatted at 2026-08-12 05:21:13 on dist-test-slave-42z9"
server_key: "72642fa7b61e9c6c68f67525e0e299d4"
server_key_iv: "9ab10b879730bb190510337cabc084ea"
server_key_version: "encryptionkey@0"
I20260812 05:21:13.522200 15721 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786512017613366-32144-0/minicluster-data/master-0/wal/instance:
uuid: "9aa2b418f50b4b828e756d747ee2cec3"
format_stamp: "Formatted at 2026-08-12 05:21:13 on dist-test-slave-42z9"
server_key: "72642fa7b61e9c6c68f67525e0e299d4"
server_key_iv: "9ab10b879730bb190510337cabc084ea"
server_key_version: "encryptionkey@0"
I20260812 05:21:13.523456 15721 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260812 05:21:13.524152 15749 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 05:21:13.524329 15721 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260812 05:21:13.524464 15721 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786512017613366-32144-0/minicluster-data/master-0/data,/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786512017613366-32144-0/minicluster-data/master-0/wal
uuid: "9aa2b418f50b4b828e756d747ee2cec3"
format_stamp: "Formatted at 2026-08-12 05:21:13 on dist-test-slave-42z9"
server_key: "72642fa7b61e9c6c68f67525e0e299d4"
server_key_iv: "9ab10b879730bb190510337cabc084ea"
server_key_version: "encryptionkey@0"
I20260812 05:21:13.524529 15721 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786512017613366-32144-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786512017613366-32144-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786512017613366-32144-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 05:21:13.527688 15721 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 05:21:13.527915 15721 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 05:21:13.528029 15721 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 05:21:13.531663 15721 rpc_server.cc:307] RPC server started. Bound to: 127.31.100.62:37181
I20260812 05:21:13.531716 15833 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.100.62:37181 every 8 connection(s)
I20260812 05:21:13.531991 15721 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786512017613366-32144-0/minicluster-data/master-0/data/info.pb
I20260812 05:21:13.532636 15835 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 05:21:13.532893 32144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0waFvT/build/release/bin/kudu as pid 15721
I20260812 05:21:13.532989 32144 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786512017613366-32144-0/minicluster-data/master-0/wal/instance
I20260812 05:21:13.533155 32144 external_mini_cluster.cc:1428] Setting key 584e058d9c34b64642dc5f0fcac8b3fe
I20260812 05:21:13.535665 15839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:21:13.533658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33122 (local address 127.31.100.62:37181)
0812 05:21:13.533867 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0812 05:21:13.533876 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 05:21:13.533917 (+ 41us) server_negotiation.cc:407] Connection header received
0812 05:21:13.534504 (+ 587us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:21:13.534513 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:21:13.534564 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:21:13.534707 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:21:13.535261 (+ 554us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 05:21:13.535265 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 05:21:13.535265 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 05:21:13.535269 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0812 05:21:13.535312 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 05:21:13.535325 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 05:21:13.535406 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0812 05:21:13.535439 (+ 33us) server_negotiation.cc:299] Negotiation successful
0812 05:21:13.535493 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":82,"threads_started":1}
I20260812 05:21:13.536193 15835 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9aa2b418f50b4b828e756d747ee2cec3: Bootstrap starting.
I20260812 05:21:13.536770 15835 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9aa2b418f50b4b828e756d747ee2cec3: Neither blocks nor log segments found. Creating new log.
I20260812 05:21:13.537034 15835 log.cc:824] T 00000000000000000000000000000000 P 9aa2b418f50b4b828e756d747ee2cec3: Log is configured to *not* fsync() on all Append() calls
I20260812 05:21:13.537699 15835 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9aa2b418f50b4b828e756d747ee2cec3: No bootstrap required, opened a new log
I20260812 05:21:13.538944 15835 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9aa2b418f50b4b828e756d747ee2cec3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9aa2b418f50b4b828e756d747ee2cec3" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 37181 } }
I20260812 05:21:13.539067 15835 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9aa2b418f50b4b828e756d747ee2cec3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 05:21:13.539088 15835 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9aa2b418f50b4b828e756d747ee2cec3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9aa2b418f50b4b828e756d747ee2cec3, State: Initialized, Role: FOLLOWER
I20260812 05:21:13.539187 15835 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9aa2b418f50b4b828e756d747ee2cec3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9aa2b418f50b4b828e756d747ee2cec3" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 37181 } }
I20260812 05:21:13.539254 15835 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9aa2b418f50b4b828e756d747ee2cec3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 05:21:13.539276 15835 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9aa2b418f50b4b828e756d747ee2cec3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 05:21:13.539322 15835 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9aa2b418f50b4b828e756d747ee2cec3 [term 0 FOLLOWER]: Advancing to term 1
I20260812 05:21:13.539801 15835 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9aa2b418f50b4b828e756d747ee2cec3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9aa2b418f50b4b828e756d747ee2cec3" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 37181 } }
I20260812 05:21:13.539889 15835 leader_election.cc:302] T 00000000000000000000000000000000 P 9aa2b418f50b4b828e756d747ee2cec3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9aa2b418f50b4b828e756d747ee2cec3; no voters:
I20260812 05:21:13.540051 15835 leader_election.cc:288] T 00000000000000000000000000000000 P 9aa2b418f50b4b828e756d747ee2cec3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 05:21:13.540299 15835 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9aa2b418f50b4b828e756d747ee2cec3 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 05:21:13.540318 15841 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9aa2b418f50b4b828e756d747ee2cec3 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 05:21:13.540477 15841 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9aa2b418f50b4b828e756d747ee2cec3 [term 1 LEADER]: Becoming Leader. State: Replica: 9aa2b418f50b4b828e756d747ee2cec3, State: Running, Role: LEADER
I20260812 05:21:13.540583 15841 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9aa2b418f50b4b828e756d747ee2cec3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9aa2b418f50b4b828e756d747ee2cec3" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 37181 } }
I20260812 05:21:13.540978 15841 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9aa2b418f50b4b828e756d747ee2cec3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9aa2b418f50b4b828e756d747ee2cec3. Latest consensus state: current_term: 1 leader_uuid: "9aa2b418f50b4b828e756d747ee2cec3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9aa2b418f50b4b828e756d747ee2cec3" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 37181 } } }
I20260812 05:21:13.541067 15841 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9aa2b418f50b4b828e756d747ee2cec3 [sys.catalog]: This master's current role is: LEADER
I20260812 05:21:13.541296 15857 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 05:21:13.541713 15842 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9aa2b418f50b4b828e756d747ee2cec3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9aa2b418f50b4b828e756d747ee2cec3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9aa2b418f50b4b828e756d747ee2cec3" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 37181 } } }
I20260812 05:21:13.541803 15842 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9aa2b418f50b4b828e756d747ee2cec3 [sys.catalog]: This master's current role is: LEADER
I20260812 05:21:13.542193 15857 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 05:21:13.543565 15857 catalog_manager.cc:1347] Generated new cluster ID: 8b2ddc84a4e1464e8364f3b6f52935fa
I20260812 05:21:13.543620 15857 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 05:21:13.554358 15857 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 05:21:13.554517 15857 catalog_manager.cc:1504] Loading token signing keys...
I20260812 05:21:13.561342 15857 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9aa2b418f50b4b828e756d747ee2cec3: Generated new TSK 0
I20260812 05:21:13.561560 15857 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 05:21:13.565755 32144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0waFvT/build/release/bin/kudu
/tmp/dist-test-task0waFvT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786512017613366-32144-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786512017613366-32144-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786512017613366-32144-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786512017613366-32144-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.100.1:0
--local_ip_for_outbound_sockets=127.31.100.1
--webserver_interface=127.31.100.1
--webserver_port=0
--tserver_master_addrs=127.31.100.62:37181
--builtin_ntp_servers=127.31.100.20:35499
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260812 05:21:13.646593 15876 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 05:21:13.646802 15876 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 05:21:13.646832 15876 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 05:21:13.647943 15876 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 05:21:13.647981 15876 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 05:21:13.648015 15876 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.100.1
I20260812 05:21:13.649325 15876 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.100.20:35499
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786512017613366-32144-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786512017613366-32144-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.100.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786512017613366-32144-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.100.1
--webserver_port=0
--tserver_master_addrs=127.31.100.62:37181
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15876
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.100.1
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786512017613366-32144-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 04:20:34 UTC on ccycloud.quasar-phqdqp.root.comops.site
build id 4581871
I20260812 05:21:13.649593 15876 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 05:21:13.649865 15876 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 05:21:13.654549 15885 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 05:21:13.654726 15876 server_base.cc:1034] running on GCE node
W20260812 05:21:13.654562 15893 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 05:21:13.654562 15887 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 05:21:13.655000 15876 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 05:21:13.655275 15876 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 05:21:13.656419 15876 hybrid_clock.cc:648] HybridClock initialized: now 1786512073656405 us; error 30 us; skew 500 ppm
I20260812 05:21:13.657609 15876 webserver.cc:460] Webserver started at http://127.31.100.1:42471/ using document root <none> and password file <none>
I20260812 05:21:13.657835 15876 fs_manager.cc:359] Metadata directory not provided
I20260812 05:21:13.657881 15876 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 05:21:13.658010 15876 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 05:21:13.659024 15876 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786512017613366-32144-0/minicluster-data/ts-0/data/instance:
uuid: "312cdd8d0ad54d44a6ce935964e74612"
format_stamp: "Formatted at 2026-08-12 05:21:13 on dist-test-slave-42z9"
server_key: "23a76f4e5cdf986ced6984e7c5ac2fe4"
server_key_iv: "9e17d49c827837bc64cdf9322e36753d"
server_key_version: "encryptionkey@0"
I20260812 05:21:13.659369 15876 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786512017613366-32144-0/minicluster-data/ts-0/wal/instance:
uuid: "312cdd8d0ad54d44a6ce935964e74612"
format_stamp: "Formatted at 2026-08-12 05:21:13 on dist-test-slave-42z9"
server_key: "23a76f4e5cdf986ced6984e7c5ac2fe4"
server_key_iv: "9e17d49c827837bc64cdf9322e36753d"
server_key_version: "encryptionkey@0"
I20260812 05:21:13.660679 15876 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260812 05:21:13.662948 15905 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 05:21:13.663110 15876 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.002s
I20260812 05:21:13.663208 15876 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786512017613366-32144-0/minicluster-data/ts-0/data,/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786512017613366-32144-0/minicluster-data/ts-0/wal
uuid: "312cdd8d0ad54d44a6ce935964e74612"
format_stamp: "Formatted at 2026-08-12 05:21:13 on dist-test-slave-42z9"
server_key: "23a76f4e5cdf986ced6984e7c5ac2fe4"
server_key_iv: "9e17d49c827837bc64cdf9322e36753d"
server_key_version: "encryptionkey@0"
I20260812 05:21:13.663276 15876 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786512017613366-32144-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786512017613366-32144-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786512017613366-32144-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 05:21:13.667675 15876 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 05:21:13.667981 15876 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 05:21:13.668124 15876 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 05:21:13.668351 15876 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 05:21:13.668697 15876 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 05:21:13.668738 15876 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 05:21:13.668768 15876 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 05:21:13.668787 15876 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 05:21:13.674777 15876 rpc_server.cc:307] RPC server started. Bound to: 127.31.100.1:44983
I20260812 05:21:13.674841 16099 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.100.1:44983 every 8 connection(s)
I20260812 05:21:13.675065 15876 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786512017613366-32144-0/minicluster-data/ts-0/data/info.pb
I20260812 05:21:13.677810 15839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:21:13.675689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.100.1:57217 (local address 127.31.100.62:37181)
0812 05:21:13.675743 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0812 05:21:13.675745 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 05:21:13.675941 (+ 196us) server_negotiation.cc:407] Connection header received
0812 05:21:13.676396 (+ 455us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:21:13.676397 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:21:13.676404 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:21:13.676456 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:21:13.677556 (+ 1100us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 05:21:13.677558 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 05:21:13.677558 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 05:21:13.677577 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0812 05:21:13.677598 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 05:21:13.677602 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 05:21:13.677645 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0812 05:21:13.677721 (+ 76us) server_negotiation.cc:299] Negotiation successful
0812 05:21:13.677736 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260812 05:21:13.677980 16107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:21:13.675681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.62:37181 (local address 127.31.100.1:57217)
0812 05:21:13.675835 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0812 05:21:13.675865 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0812 05:21:13.676306 (+ 441us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:21:13.676517 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:21:13.676520 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:21:13.676527 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0812 05:21:13.677379 (+ 852us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 05:21:13.677380 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0812 05:21:13.677407 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 05:21:13.677433 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 05:21:13.677650 (+ 217us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 05:21:13.677656 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 05:21:13.677659 (+ 3us) client_negotiation.cc:769] Sending connection context
0812 05:21:13.677728 (+ 69us) client_negotiation.cc:240] Negotiation successful
0812 05:21:13.677786 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":69,"threads_started":1}
I20260812 05:21:13.678419 16105 heartbeater.cc:347] Connected to a master server at 127.31.100.62:37181
I20260812 05:21:13.678498 16105 heartbeater.cc:464] Registering TS with master...
I20260812 05:21:13.678637 16105 heartbeater.cc:511] Master 127.31.100.62:37181 requested a full tablet report, sending...
I20260812 05:21:13.679090 15775 ts_manager.cc:194] Registered new tserver with Master: 312cdd8d0ad54d44a6ce935964e74612 (127.31.100.1:44983)
I20260812 05:21:13.683285 32144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0waFvT/build/release/bin/kudu as pid 15876
I20260812 05:21:13.683369 32144 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786512017613366-32144-0/minicluster-data/ts-0/wal/instance
I20260812 05:21:13.683555 32144 external_mini_cluster.cc:1428] Setting key 098d456476f5b246c743aecdef8605ce
I20260812 05:21:13.684358 32144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0waFvT/build/release/bin/kudu
/tmp/dist-test-task0waFvT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786512017613366-32144-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786512017613366-32144-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786512017613366-32144-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786512017613366-32144-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.100.2:0
--local_ip_for_outbound_sockets=127.31.100.2
--webserver_interface=127.31.100.2
--webserver_port=0
--tserver_master_addrs=127.31.100.62:37181
--builtin_ntp_servers=127.31.100.20:35499
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260812 05:21:13.763712 16108 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 05:21:13.763898 16108 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 05:21:13.763919 16108 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 05:21:13.765041 16108 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 05:21:13.765080 16108 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 05:21:13.765105 16108 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.100.2
I20260812 05:21:13.766417 16108 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.100.20:35499
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786512017613366-32144-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786512017613366-32144-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.100.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786512017613366-32144-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.100.2
--webserver_port=0
--tserver_master_addrs=127.31.100.62:37181
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16108
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.100.2
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786512017613366-32144-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 04:20:34 UTC on ccycloud.quasar-phqdqp.root.comops.site
build id 4581871
I20260812 05:21:13.766638 16108 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 05:21:13.766933 16108 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 05:21:13.771752 16117 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 05:21:13.771737 16118 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 05:21:13.771883 16108 server_base.cc:1034] running on GCE node
W20260812 05:21:13.771809 16123 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 05:21:13.772286 16108 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 05:21:13.772523 16108 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 05:21:13.773662 16108 hybrid_clock.cc:648] HybridClock initialized: now 1786512073773650 us; error 30 us; skew 500 ppm
I20260812 05:21:13.774873 16108 webserver.cc:460] Webserver started at http://127.31.100.2:36867/ using document root <none> and password file <none>
I20260812 05:21:13.775099 16108 fs_manager.cc:359] Metadata directory not provided
I20260812 05:21:13.775149 16108 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 05:21:13.775272 16108 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 05:21:13.776262 16108 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786512017613366-32144-0/minicluster-data/ts-1/data/instance:
uuid: "6dee33e4cb46456bb2b352d4a010cb62"
format_stamp: "Formatted at 2026-08-12 05:21:13 on dist-test-slave-42z9"
server_key: "e0ce59fa1c56fe144a5dcc8356e0727c"
server_key_iv: "5ad3ce7b3f36bbbf5f80de1d78a1b415"
server_key_version: "encryptionkey@0"
I20260812 05:21:13.776587 16108 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786512017613366-32144-0/minicluster-data/ts-1/wal/instance:
uuid: "6dee33e4cb46456bb2b352d4a010cb62"
format_stamp: "Formatted at 2026-08-12 05:21:13 on dist-test-slave-42z9"
server_key: "e0ce59fa1c56fe144a5dcc8356e0727c"
server_key_iv: "5ad3ce7b3f36bbbf5f80de1d78a1b415"
server_key_version: "encryptionkey@0"
I20260812 05:21:13.777810 16108 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260812 05:21:13.778432 16134 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 05:21:13.778630 16108 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260812 05:21:13.778725 16108 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786512017613366-32144-0/minicluster-data/ts-1/data,/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786512017613366-32144-0/minicluster-data/ts-1/wal
uuid: "6dee33e4cb46456bb2b352d4a010cb62"
format_stamp: "Formatted at 2026-08-12 05:21:13 on dist-test-slave-42z9"
server_key: "e0ce59fa1c56fe144a5dcc8356e0727c"
server_key_iv: "5ad3ce7b3f36bbbf5f80de1d78a1b415"
server_key_version: "encryptionkey@0"
I20260812 05:21:13.778789 16108 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786512017613366-32144-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786512017613366-32144-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786512017613366-32144-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 05:21:13.781919 16108 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 05:21:13.782119 16108 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 05:21:13.782207 16108 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 05:21:13.782363 16108 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 05:21:13.782601 16108 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 05:21:13.782630 16108 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 05:21:13.782655 16108 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 05:21:13.782670 16108 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 05:21:13.787968 16108 rpc_server.cc:307] RPC server started. Bound to: 127.31.100.2:43397
I20260812 05:21:13.788019 16338 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.100.2:43397 every 8 connection(s)
I20260812 05:21:13.788295 16108 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786512017613366-32144-0/minicluster-data/ts-1/data/info.pb
I20260812 05:21:13.789244 32144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0waFvT/build/release/bin/kudu as pid 16108
I20260812 05:21:13.789340 32144 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786512017613366-32144-0/minicluster-data/ts-1/wal/instance
I20260812 05:21:13.789553 32144 external_mini_cluster.cc:1428] Setting key cae473d0367cd43e6077e6a97cca5856
I20260812 05:21:13.790501 32144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0waFvT/build/release/bin/kudu
/tmp/dist-test-task0waFvT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786512017613366-32144-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786512017613366-32144-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786512017613366-32144-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786512017613366-32144-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.100.3:0
--local_ip_for_outbound_sockets=127.31.100.3
--webserver_interface=127.31.100.3
--webserver_port=0
--tserver_master_addrs=127.31.100.62:37181
--builtin_ntp_servers=127.31.100.20:35499
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260812 05:21:13.791375 15839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:21:13.789044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.100.2:41921 (local address 127.31.100.62:37181)
0812 05:21:13.789127 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0812 05:21:13.789129 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 05:21:13.789346 (+ 217us) server_negotiation.cc:407] Connection header received
0812 05:21:13.789907 (+ 561us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:21:13.789908 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:21:13.789914 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:21:13.789952 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:21:13.791134 (+ 1182us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 05:21:13.791135 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 05:21:13.791135 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 05:21:13.791140 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0812 05:21:13.791157 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 05:21:13.791161 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 05:21:13.791211 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0812 05:21:13.791290 (+ 79us) server_negotiation.cc:299] Negotiation successful
0812 05:21:13.791305 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260812 05:21:13.791553 16343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:21:13.788960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.62:37181 (local address 127.31.100.2:41921)
0812 05:21:13.789216 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0812 05:21:13.789249 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0812 05:21:13.789815 (+ 566us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:21:13.790027 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:21:13.790030 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:21:13.790040 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0812 05:21:13.790994 (+ 954us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 05:21:13.790995 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0812 05:21:13.791022 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 05:21:13.791050 (+ 28us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 05:21:13.791214 (+ 164us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 05:21:13.791219 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 05:21:13.791222 (+ 3us) client_negotiation.cc:769] Sending connection context
0812 05:21:13.791292 (+ 70us) client_negotiation.cc:240] Negotiation successful
0812 05:21:13.791352 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":95,"threads_started":1}
I20260812 05:21:13.792346 16341 heartbeater.cc:347] Connected to a master server at 127.31.100.62:37181
I20260812 05:21:13.792436 16341 heartbeater.cc:464] Registering TS with master...
I20260812 05:21:13.792572 16341 heartbeater.cc:511] Master 127.31.100.62:37181 requested a full tablet report, sending...
I20260812 05:21:13.792917 15775 ts_manager.cc:194] Registered new tserver with Master: 6dee33e4cb46456bb2b352d4a010cb62 (127.31.100.2:43397)
W20260812 05:21:13.872169 16345 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 05:21:13.872367 16345 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 05:21:13.872397 16345 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 05:21:13.873546 16345 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 05:21:13.873584 16345 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 05:21:13.873617 16345 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.100.3
I20260812 05:21:13.874939 16345 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.100.20:35499
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786512017613366-32144-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786512017613366-32144-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.100.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786512017613366-32144-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.100.3
--webserver_port=0
--tserver_master_addrs=127.31.100.62:37181
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16345
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.100.3
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786512017613366-32144-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 04:20:34 UTC on ccycloud.quasar-phqdqp.root.comops.site
build id 4581871
I20260812 05:21:13.875160 16345 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 05:21:13.875422 16345 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 05:21:13.880126 16355 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 05:21:13.880133 16357 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 05:21:13.880146 16359 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 05:21:13.880375 16345 server_base.cc:1034] running on GCE node
I20260812 05:21:13.880599 16345 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 05:21:13.880779 16345 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 05:21:13.881915 16345 hybrid_clock.cc:648] HybridClock initialized: now 1786512073881899 us; error 32 us; skew 500 ppm
I20260812 05:21:13.883093 16345 webserver.cc:460] Webserver started at http://127.31.100.3:45773/ using document root <none> and password file <none>
I20260812 05:21:13.883301 16345 fs_manager.cc:359] Metadata directory not provided
I20260812 05:21:13.883343 16345 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 05:21:13.883459 16345 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 05:21:13.884465 16345 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786512017613366-32144-0/minicluster-data/ts-2/data/instance:
uuid: "510c0fcee4ef467a8607ee000d0fb378"
format_stamp: "Formatted at 2026-08-12 05:21:13 on dist-test-slave-42z9"
server_key: "bc768786ee701733d711f7b4df5ae44f"
server_key_iv: "077519d8a4c65528252c03020adfdb32"
server_key_version: "encryptionkey@0"
I20260812 05:21:13.884778 16345 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786512017613366-32144-0/minicluster-data/ts-2/wal/instance:
uuid: "510c0fcee4ef467a8607ee000d0fb378"
format_stamp: "Formatted at 2026-08-12 05:21:13 on dist-test-slave-42z9"
server_key: "bc768786ee701733d711f7b4df5ae44f"
server_key_iv: "077519d8a4c65528252c03020adfdb32"
server_key_version: "encryptionkey@0"
I20260812 05:21:13.886087 16345 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260812 05:21:13.886785 16371 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 05:21:13.886956 16345 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260812 05:21:13.887041 16345 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786512017613366-32144-0/minicluster-data/ts-2/data,/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786512017613366-32144-0/minicluster-data/ts-2/wal
uuid: "510c0fcee4ef467a8607ee000d0fb378"
format_stamp: "Formatted at 2026-08-12 05:21:13 on dist-test-slave-42z9"
server_key: "bc768786ee701733d711f7b4df5ae44f"
server_key_iv: "077519d8a4c65528252c03020adfdb32"
server_key_version: "encryptionkey@0"
I20260812 05:21:13.887099 16345 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786512017613366-32144-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786512017613366-32144-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786512017613366-32144-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 05:21:13.890337 16345 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 05:21:13.890568 16345 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 05:21:13.890681 16345 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 05:21:13.890863 16345 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 05:21:13.891139 16345 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 05:21:13.891172 16345 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 05:21:13.891197 16345 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 05:21:13.891220 16345 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 05:21:13.896526 16345 rpc_server.cc:307] RPC server started. Bound to: 127.31.100.3:36037
I20260812 05:21:13.896584 16565 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.100.3:36037 every 8 connection(s)
I20260812 05:21:13.896847 16345 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786512017613366-32144-0/minicluster-data/ts-2/data/info.pb
I20260812 05:21:13.899506 15839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:21:13.897461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.100.3:56241 (local address 127.31.100.62:37181)
0812 05:21:13.897506 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0812 05:21:13.897508 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 05:21:13.897718 (+ 210us) server_negotiation.cc:407] Connection header received
0812 05:21:13.898187 (+ 469us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:21:13.898188 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:21:13.898196 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:21:13.898252 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:21:13.899260 (+ 1008us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 05:21:13.899261 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 05:21:13.899261 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 05:21:13.899266 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0812 05:21:13.899282 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 05:21:13.899286 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 05:21:13.899336 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0812 05:21:13.899417 (+ 81us) server_negotiation.cc:299] Negotiation successful
0812 05:21:13.899438 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260812 05:21:13.899672 16569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:21:13.897428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.62:37181 (local address 127.31.100.3:56241)
0812 05:21:13.897612 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0812 05:21:13.897641 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0812 05:21:13.898087 (+ 446us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:21:13.898317 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:21:13.898321 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:21:13.898331 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0812 05:21:13.899123 (+ 792us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 05:21:13.899124 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0812 05:21:13.899150 (+ 26us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 05:21:13.899175 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 05:21:13.899339 (+ 164us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 05:21:13.899344 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 05:21:13.899347 (+ 3us) client_negotiation.cc:769] Sending connection context
0812 05:21:13.899423 (+ 76us) client_negotiation.cc:240] Negotiation successful
0812 05:21:13.899482 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":75,"threads_started":1}
I20260812 05:21:13.900076 16567 heartbeater.cc:347] Connected to a master server at 127.31.100.62:37181
I20260812 05:21:13.900154 16567 heartbeater.cc:464] Registering TS with master...
I20260812 05:21:13.900288 16567 heartbeater.cc:511] Master 127.31.100.62:37181 requested a full tablet report, sending...
I20260812 05:21:13.900624 15775 ts_manager.cc:194] Registered new tserver with Master: 510c0fcee4ef467a8607ee000d0fb378 (127.31.100.3:36037)
I20260812 05:21:13.905859 32144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0waFvT/build/release/bin/kudu as pid 16345
I20260812 05:21:13.905943 32144 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786512017613366-32144-0/minicluster-data/ts-2/wal/instance
I20260812 05:21:13.906086 32144 external_mini_cluster.cc:1428] Setting key 965cadacc45a3d19fd3bdd9ef570ce65
I20260812 05:21:13.906857 32144 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260812 05:21:13.908545 15839 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 05:21:13.908075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33138 (local address 127.31.100.62:37181)
0812 05:21:13.908117 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0812 05:21:13.908119 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 05:21:13.908243 (+ 124us) server_negotiation.cc:407] Connection header received
0812 05:21:13.908285 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:21:13.908286 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:21:13.908290 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:21:13.908334 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:21:13.908487 (+ 153us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33138: BlockingRecv error: recv got EOF from 127.0.0.1:33138 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
I20260812 05:21:13.908567 32144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0waFvT/build/release/bin/kudu with pid 15876
I20260812 05:21:13.916646 32144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0waFvT/build/release/bin/kudu with pid 16108
I20260812 05:21:13.926630 32144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0waFvT/build/release/bin/kudu with pid 16345
I20260812 05:21:13.932310 32144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0waFvT/build/release/bin/kudu with pid 15721
2026-08-12T05:21:13Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (537 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786512017613366-32144-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 05:21:13 dist-test-slave-42z9 krb5kdc[16582](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 05:21:13 dist-test-slave-42z9 krb5kdc[16582](info): set up 2 sockets
Aug 12 05:21:13 dist-test-slave-42z9 krb5kdc[16582](info): commencing operation
krb5kdc: starting...
W20260812 05:21:15.112108 32144 mini_kdc.cc:121] Time spent starting KDC: real 1.153s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786512017613366-32144-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786512017613366-32144-0/krb5kdc/test-user.keytab.
Aug 12 05:21:15 dist-test-slave-42z9 krb5kdc[16582](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512075, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T05:21:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T05:21:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.100.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.100.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.100.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.62.keytab.
Entry for principal kudu/127.31.100.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.62.keytab.
I20260812 05:21:15.230250 32144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0waFvT/build/release/bin/kudu
/tmp/dist-test-task0waFvT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786512017613366-32144-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786512017613366-32144-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786512017613366-32144-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786512017613366-32144-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.100.62:37923
--webserver_interface=127.31.100.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.62.keytab
--principal=kudu/127.31.100.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.100.20:46431
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.100.62:37923
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786512017613366-32144-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786512017613366-32144-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786512017613366-32144-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 05:21:15.310319 16605 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 05:21:15.310519 16605 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 05:21:15.310539 16605 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 05:21:15.311638 16605 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 05:21:15.311678 16605 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 05:21:15.311690 16605 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 05:21:15.311713 16605 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 05:21:15.311735 16605 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 05:21:15.313218 16605 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.100.20:46431
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786512017613366-32144-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786512017613366-32144-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.100.62:37923
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.62.keytab
--principal=kudu/127.31.100.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.100.62:37923
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786512017613366-32144-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.100.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16605
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786512017613366-32144-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 04:20:34 UTC on ccycloud.quasar-phqdqp.root.comops.site
build id 4581871
I20260812 05:21:15.313479 16605 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 05:21:15.313684 16605 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 05:21:15.318291 16616 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 05:21:15.318267 16621 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 05:21:15.318284 16617 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 05:21:15.318495 16605 server_base.cc:1034] running on GCE node
I20260812 05:21:15.318737 16605 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 05:21:15.319000 16605 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 05:21:15.320148 16605 hybrid_clock.cc:648] HybridClock initialized: now 1786512075320132 us; error 27 us; skew 500 ppm
Aug 12 05:21:15 dist-test-slave-42z9 krb5kdc[16582](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512075, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.100.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 05:21:15.322459 16605 init.cc:377] Logged in from keytab as kudu/127.31.100.62@KRBTEST.COM (short username kudu)
I20260812 05:21:15.322829 16605 webserver.cc:460] Webserver started at http://127.31.100.62:45395/ using document root <none> and password file <none>
I20260812 05:21:15.323009 16605 fs_manager.cc:359] Metadata directory not provided
I20260812 05:21:15.323047 16605 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 05:21:15.323143 16605 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 05:21:15.324083 16605 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786512017613366-32144-0/minicluster-data/master-0/data/instance:
uuid: "fd0e2476c9b1458fb2a281641ae24e52"
format_stamp: "Formatted at 2026-08-12 05:21:15 on dist-test-slave-42z9"
server_key: "bfad3423128a1c6e6717bc3e68b4020d"
server_key_iv: "c2b49e152537240668042bc7d99b1ca7"
server_key_version: "encryptionkey@0"
I20260812 05:21:15.324388 16605 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786512017613366-32144-0/minicluster-data/master-0/wal/instance:
uuid: "fd0e2476c9b1458fb2a281641ae24e52"
format_stamp: "Formatted at 2026-08-12 05:21:15 on dist-test-slave-42z9"
server_key: "bfad3423128a1c6e6717bc3e68b4020d"
server_key_iv: "c2b49e152537240668042bc7d99b1ca7"
server_key_version: "encryptionkey@0"
I20260812 05:21:15.325626 16605 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260812 05:21:15.326256 16628 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 05:21:15.326383 16605 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260812 05:21:15.326457 16605 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786512017613366-32144-0/minicluster-data/master-0/data,/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786512017613366-32144-0/minicluster-data/master-0/wal
uuid: "fd0e2476c9b1458fb2a281641ae24e52"
format_stamp: "Formatted at 2026-08-12 05:21:15 on dist-test-slave-42z9"
server_key: "bfad3423128a1c6e6717bc3e68b4020d"
server_key_iv: "c2b49e152537240668042bc7d99b1ca7"
server_key_version: "encryptionkey@0"
I20260812 05:21:15.326521 16605 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786512017613366-32144-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786512017613366-32144-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786512017613366-32144-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 05:21:15.344492 16605 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 05:21:15.346244 16605 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 05:21:15.346370 16605 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 05:21:15.350154 16605 rpc_server.cc:307] RPC server started. Bound to: 127.31.100.62:37923
I20260812 05:21:15.350189 16732 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.100.62:37923 every 8 connection(s)
I20260812 05:21:15.350507 16605 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786512017613366-32144-0/minicluster-data/master-0/data/info.pb
I20260812 05:21:15.351502 16735 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 05:21:15.353732 16735 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd0e2476c9b1458fb2a281641ae24e52: Bootstrap starting.
I20260812 05:21:15.354260 16735 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fd0e2476c9b1458fb2a281641ae24e52: Neither blocks nor log segments found. Creating new log.
I20260812 05:21:15.354499 16735 log.cc:824] T 00000000000000000000000000000000 P fd0e2476c9b1458fb2a281641ae24e52: Log is configured to *not* fsync() on all Append() calls
I20260812 05:21:15.354736 32144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0waFvT/build/release/bin/kudu as pid 16605
I20260812 05:21:15.354818 32144 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786512017613366-32144-0/minicluster-data/master-0/wal/instance
I20260812 05:21:15.354997 32144 external_mini_cluster.cc:1428] Setting key 95871e0938a036444d3d9614429e2827
I20260812 05:21:15.355250 16735 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd0e2476c9b1458fb2a281641ae24e52: No bootstrap required, opened a new log
I20260812 05:21:15.356979 16735 raft_consensus.cc:348] T 00000000000000000000000000000000 P fd0e2476c9b1458fb2a281641ae24e52 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd0e2476c9b1458fb2a281641ae24e52" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 37923 } }
I20260812 05:21:15.357117 16735 raft_consensus.cc:374] T 00000000000000000000000000000000 P fd0e2476c9b1458fb2a281641ae24e52 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 05:21:15.357148 16735 raft_consensus.cc:729] T 00000000000000000000000000000000 P fd0e2476c9b1458fb2a281641ae24e52 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd0e2476c9b1458fb2a281641ae24e52, State: Initialized, Role: FOLLOWER
I20260812 05:21:15.357291 16735 consensus_queue.cc:260] T 00000000000000000000000000000000 P fd0e2476c9b1458fb2a281641ae24e52 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd0e2476c9b1458fb2a281641ae24e52" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 37923 } }
I20260812 05:21:15.357362 16735 raft_consensus.cc:388] T 00000000000000000000000000000000 P fd0e2476c9b1458fb2a281641ae24e52 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 05:21:15.357391 16735 raft_consensus.cc:482] T 00000000000000000000000000000000 P fd0e2476c9b1458fb2a281641ae24e52 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 05:21:15.357461 16735 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fd0e2476c9b1458fb2a281641ae24e52 [term 0 FOLLOWER]: Advancing to term 1
I20260812 05:21:15.358246 16735 raft_consensus.cc:504] T 00000000000000000000000000000000 P fd0e2476c9b1458fb2a281641ae24e52 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd0e2476c9b1458fb2a281641ae24e52" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 37923 } }
I20260812 05:21:15.358343 16735 leader_election.cc:302] T 00000000000000000000000000000000 P fd0e2476c9b1458fb2a281641ae24e52 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fd0e2476c9b1458fb2a281641ae24e52; no voters:
I20260812 05:21:15.358493 16735 leader_election.cc:288] T 00000000000000000000000000000000 P fd0e2476c9b1458fb2a281641ae24e52 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 05:21:15.358526 16742 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fd0e2476c9b1458fb2a281641ae24e52 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 05:21:15.358639 16742 raft_consensus.cc:686] T 00000000000000000000000000000000 P fd0e2476c9b1458fb2a281641ae24e52 [term 1 LEADER]: Becoming Leader. State: Replica: fd0e2476c9b1458fb2a281641ae24e52, State: Running, Role: LEADER
I20260812 05:21:15.358750 16735 sys_catalog.cc:564] T 00000000000000000000000000000000 P fd0e2476c9b1458fb2a281641ae24e52 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 05:21:15.358780 16742 consensus_queue.cc:237] T 00000000000000000000000000000000 P fd0e2476c9b1458fb2a281641ae24e52 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd0e2476c9b1458fb2a281641ae24e52" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 37923 } }
I20260812 05:21:15.359151 16744 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd0e2476c9b1458fb2a281641ae24e52 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fd0e2476c9b1458fb2a281641ae24e52" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd0e2476c9b1458fb2a281641ae24e52" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 37923 } } }
I20260812 05:21:15.359163 16743 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd0e2476c9b1458fb2a281641ae24e52 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fd0e2476c9b1458fb2a281641ae24e52. Latest consensus state: current_term: 1 leader_uuid: "fd0e2476c9b1458fb2a281641ae24e52" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd0e2476c9b1458fb2a281641ae24e52" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 37923 } } }
I20260812 05:21:15.359246 16744 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd0e2476c9b1458fb2a281641ae24e52 [sys.catalog]: This master's current role is: LEADER
I20260812 05:21:15.359265 16743 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd0e2476c9b1458fb2a281641ae24e52 [sys.catalog]: This master's current role is: LEADER
I20260812 05:21:15.360152 16748 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 05:21:15.361335 16748 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Aug 12 05:21:15 dist-test-slave-42z9 krb5kdc[16582](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512075, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.100.62@KRBTEST.COM
I20260812 05:21:15.362916 16748 catalog_manager.cc:1347] Generated new cluster ID: 6c26edb7083e4e46bcb5f3d45f0c9945
I20260812 05:21:15.362958 16748 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 05:21:15.367172 16741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:21:15.355475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35898 (local address 127.31.100.62:37923)
0812 05:21:15.355661 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0812 05:21:15.355669 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 05:21:15.355695 (+ 26us) server_negotiation.cc:407] Connection header received
0812 05:21:15.356990 (+ 1295us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:21:15.356999 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:21:15.357053 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:21:15.357202 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:21:15.357920 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:21:15.359036 (+ 1116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:21:15.360140 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:21:15.360610 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:21:15.360691 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:21:15.363379 (+ 2688us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 05:21:15.363382 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 05:21:15.363383 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 05:21:15.363393 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0812 05:21:15.365187 (+ 1794us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:21:15.365586 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:21:15.365589 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:21:15.365589 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:21:15.365720 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:21:15.365914 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:21:15.365915 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:21:15.365915 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:21:15.366188 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 05:21:15.366772 (+ 584us) server_negotiation.cc:1035] Waiting for connection context
0812 05:21:15.366841 (+ 69us) server_negotiation.cc:299] Negotiation successful
0812 05:21:15.366975 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":92,"threads_started":1}
I20260812 05:21:15.384570 16748 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 05:21:15.385105 16748 catalog_manager.cc:1504] Loading token signing keys...
I20260812 05:21:15.387486 16748 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fd0e2476c9b1458fb2a281641ae24e52: Generated new TSK 0
I20260812 05:21:15.387647 16748 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.31.100.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.100.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.100.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.1.keytab.
Entry for principal kudu/127.31.100.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.1.keytab.
I20260812 05:21:15.419564 32144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0waFvT/build/release/bin/kudu
/tmp/dist-test-task0waFvT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786512017613366-32144-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786512017613366-32144-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786512017613366-32144-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786512017613366-32144-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.100.1:0
--local_ip_for_outbound_sockets=127.31.100.1
--webserver_interface=127.31.100.1
--webserver_port=0
--tserver_master_addrs=127.31.100.62:37923
--keytab_file=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.1.keytab
--principal=kudu/127.31.100.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.100.20:46431
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786512017613366-32144-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786512017613366-32144-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786512017613366-32144-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 05:21:15.499614 16775 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 05:21:15.499802 16775 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 05:21:15.499825 16775 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 05:21:15.500866 16775 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 05:21:15.500905 16775 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 05:21:15.500931 16775 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.100.1
I20260812 05:21:15.502293 16775 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.100.20:46431
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786512017613366-32144-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786512017613366-32144-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.1.keytab
--principal=kudu/127.31.100.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.100.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786512017613366-32144-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.100.1
--webserver_port=0
--tserver_master_addrs=127.31.100.62:37923
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16775
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.100.1
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786512017613366-32144-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 04:20:34 UTC on ccycloud.quasar-phqdqp.root.comops.site
build id 4581871
I20260812 05:21:15.502467 16775 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 05:21:15.502638 16775 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 05:21:15.507326 16793 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 05:21:15.507339 16787 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 05:21:15.507390 16796 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 05:21:15.507366 16775 server_base.cc:1034] running on GCE node
I20260812 05:21:15.507726 16775 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 05:21:15.507966 16775 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 05:21:15.509122 16775 hybrid_clock.cc:648] HybridClock initialized: now 1786512075509107 us; error 27 us; skew 500 ppm
Aug 12 05:21:15 dist-test-slave-42z9 krb5kdc[16582](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512075, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.100.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 05:21:15.511493 16775 init.cc:377] Logged in from keytab as kudu/127.31.100.1@KRBTEST.COM (short username kudu)
I20260812 05:21:15.511876 16775 webserver.cc:460] Webserver started at http://127.31.100.1:45759/ using document root <none> and password file <none>
I20260812 05:21:15.512063 16775 fs_manager.cc:359] Metadata directory not provided
I20260812 05:21:15.512107 16775 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 05:21:15.512226 16775 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 05:21:15.513175 16775 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786512017613366-32144-0/minicluster-data/ts-0/data/instance:
uuid: "949f45ad4d304bde99cf4bc402002bcb"
format_stamp: "Formatted at 2026-08-12 05:21:15 on dist-test-slave-42z9"
server_key: "4fc6c91e3afdde0f5e44e3da68f8a333"
server_key_iv: "b8a208fa715721148a99d461946899e0"
server_key_version: "encryptionkey@0"
I20260812 05:21:15.513535 16775 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786512017613366-32144-0/minicluster-data/ts-0/wal/instance:
uuid: "949f45ad4d304bde99cf4bc402002bcb"
format_stamp: "Formatted at 2026-08-12 05:21:15 on dist-test-slave-42z9"
server_key: "4fc6c91e3afdde0f5e44e3da68f8a333"
server_key_iv: "b8a208fa715721148a99d461946899e0"
server_key_version: "encryptionkey@0"
I20260812 05:21:15.514748 16775 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260812 05:21:15.515429 16809 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 05:21:15.515564 16775 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260812 05:21:15.515647 16775 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786512017613366-32144-0/minicluster-data/ts-0/data,/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786512017613366-32144-0/minicluster-data/ts-0/wal
uuid: "949f45ad4d304bde99cf4bc402002bcb"
format_stamp: "Formatted at 2026-08-12 05:21:15 on dist-test-slave-42z9"
server_key: "4fc6c91e3afdde0f5e44e3da68f8a333"
server_key_iv: "b8a208fa715721148a99d461946899e0"
server_key_version: "encryptionkey@0"
I20260812 05:21:15.515710 16775 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786512017613366-32144-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786512017613366-32144-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786512017613366-32144-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 05:21:15.546360 16775 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 05:21:15.548282 16775 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 05:21:15.548429 16775 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 05:21:15.548659 16775 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 05:21:15.548961 16775 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 05:21:15.548995 16775 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 05:21:15.549026 16775 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 05:21:15.549048 16775 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 05:21:15.554323 16775 rpc_server.cc:307] RPC server started. Bound to: 127.31.100.1:41991
I20260812 05:21:15.554397 17008 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.100.1:41991 every 8 connection(s)
I20260812 05:21:15.554682 16775 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786512017613366-32144-0/minicluster-data/ts-0/data/info.pb
Aug 12 05:21:15 dist-test-slave-42z9 krb5kdc[16582](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512075, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.100.1@KRBTEST.COM for kudu/127.31.100.62@KRBTEST.COM
I20260812 05:21:15.563266 16741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:21:15.555355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.100.1:36373 (local address 127.31.100.62:37923)
0812 05:21:15.555440 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0812 05:21:15.555441 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0812 05:21:15.555606 (+ 165us) server_negotiation.cc:407] Connection header received
0812 05:21:15.556086 (+ 480us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:21:15.556087 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:21:15.556099 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:21:15.556151 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:21:15.557024 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:21:15.557588 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:21:15.558491 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:21:15.558742 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:21:15.558793 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:21:15.561002 (+ 2209us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 05:21:15.561003 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 05:21:15.561003 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 05:21:15.561010 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 05:21:15.562218 (+ 1208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:21:15.562571 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:21:15.562572 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:21:15.562572 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:21:15.562620 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:21:15.562796 (+ 176us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:21:15.562797 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:21:15.562797 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:21:15.562886 (+ 89us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 05:21:15.562945 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0812 05:21:15.563141 (+ 196us) server_negotiation.cc:299] Negotiation successful
0812 05:21:15.563203 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260812 05:21:15.563411 17011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:21:15.555321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.62:37923 (local address 127.31.100.1:36373)
0812 05:21:15.555486 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0812 05:21:15.555527 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0812 05:21:15.556011 (+ 484us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:21:15.556204 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:21:15.556208 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:21:15.556767 (+ 559us) client_negotiation.cc:189] Negotiated authn=SASL
0812 05:21:15.556955 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:21:15.556960 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:21:15.557648 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:21:15.557649 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:21:15.558430 (+ 781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:21:15.558432 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:21:15.558810 (+ 378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:21:15.558811 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:21:15.558915 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:21:15.559408 (+ 493us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 05:21:15.559444 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0812 05:21:15.560911 (+ 1467us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 05:21:15.562286 (+ 1375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 05:21:15.562292 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 05:21:15.562294 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 05:21:15.562513 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 05:21:15.562666 (+ 153us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 05:21:15.562666 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 05:21:15.562666 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 05:21:15.562755 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 05:21:15.562940 (+ 185us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 05:21:15.562943 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 05:21:15.563047 (+ 104us) client_negotiation.cc:769] Sending connection context
0812 05:21:15.563121 (+ 74us) client_negotiation.cc:240] Negotiation successful
0812 05:21:15.563232 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":72,"threads_started":1}
I20260812 05:21:15.563863 17009 heartbeater.cc:347] Connected to a master server at 127.31.100.62:37923
I20260812 05:21:15.563954 17009 heartbeater.cc:464] Registering TS with master...
I20260812 05:21:15.564119 17009 heartbeater.cc:511] Master 127.31.100.62:37923 requested a full tablet report, sending...
I20260812 05:21:15.564512 32144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0waFvT/build/release/bin/kudu as pid 16775
I20260812 05:21:15.564641 32144 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786512017613366-32144-0/minicluster-data/ts-0/wal/instance
I20260812 05:21:15.564661 16655 ts_manager.cc:194] Registered new tserver with Master: 949f45ad4d304bde99cf4bc402002bcb (127.31.100.1:41991)
I20260812 05:21:15.564870 32144 external_mini_cluster.cc:1428] Setting key 65ece33410d7f425746ec9f042d28919
I20260812 05:21:15.565510 16655 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.100.1@KRBTEST.COM'} at 127.31.100.1:36373
WARNING: no policy specified for kudu/127.31.100.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.100.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.100.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.2.keytab.
Entry for principal kudu/127.31.100.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.2.keytab.
I20260812 05:21:15.589216 32144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0waFvT/build/release/bin/kudu
/tmp/dist-test-task0waFvT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786512017613366-32144-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786512017613366-32144-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786512017613366-32144-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786512017613366-32144-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.100.2:0
--local_ip_for_outbound_sockets=127.31.100.2
--webserver_interface=127.31.100.2
--webserver_port=0
--tserver_master_addrs=127.31.100.62:37923
--keytab_file=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.2.keytab
--principal=kudu/127.31.100.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.100.20:46431
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786512017613366-32144-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786512017613366-32144-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786512017613366-32144-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 05:21:15.668963 17016 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 05:21:15.669159 17016 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 05:21:15.669189 17016 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 05:21:15.670312 17016 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 05:21:15.670351 17016 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 05:21:15.670393 17016 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.100.2
I20260812 05:21:15.671815 17016 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.100.20:46431
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786512017613366-32144-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786512017613366-32144-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.2.keytab
--principal=kudu/127.31.100.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.100.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786512017613366-32144-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.100.2
--webserver_port=0
--tserver_master_addrs=127.31.100.62:37923
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17016
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.100.2
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786512017613366-32144-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 04:20:34 UTC on ccycloud.quasar-phqdqp.root.comops.site
build id 4581871
I20260812 05:21:15.672050 17016 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 05:21:15.672253 17016 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 05:21:15.677022 17034 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 05:21:15.677120 17016 server_base.cc:1034] running on GCE node
W20260812 05:21:15.677136 17028 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 05:21:15.677049 17039 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 05:21:15.677410 17016 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 05:21:15.677675 17016 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 05:21:15.678812 17016 hybrid_clock.cc:648] HybridClock initialized: now 1786512075678800 us; error 27 us; skew 500 ppm
Aug 12 05:21:15 dist-test-slave-42z9 krb5kdc[16582](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512075, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.100.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 05:21:15.681143 17016 init.cc:377] Logged in from keytab as kudu/127.31.100.2@KRBTEST.COM (short username kudu)
I20260812 05:21:15.681613 17016 webserver.cc:460] Webserver started at http://127.31.100.2:44451/ using document root <none> and password file <none>
I20260812 05:21:15.681815 17016 fs_manager.cc:359] Metadata directory not provided
I20260812 05:21:15.681861 17016 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 05:21:15.681998 17016 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 05:21:15.682966 17016 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786512017613366-32144-0/minicluster-data/ts-1/data/instance:
uuid: "ce209428367c4501b2008b986015ee4f"
format_stamp: "Formatted at 2026-08-12 05:21:15 on dist-test-slave-42z9"
server_key: "22d021ddf760ce60cdca6b111a9ed6f0"
server_key_iv: "407fe8154c028c7e3552616b0c66edb0"
server_key_version: "encryptionkey@0"
I20260812 05:21:15.683292 17016 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786512017613366-32144-0/minicluster-data/ts-1/wal/instance:
uuid: "ce209428367c4501b2008b986015ee4f"
format_stamp: "Formatted at 2026-08-12 05:21:15 on dist-test-slave-42z9"
server_key: "22d021ddf760ce60cdca6b111a9ed6f0"
server_key_iv: "407fe8154c028c7e3552616b0c66edb0"
server_key_version: "encryptionkey@0"
I20260812 05:21:15.684486 17016 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260812 05:21:15.685156 17050 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 05:21:15.685309 17016 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260812 05:21:15.685395 17016 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786512017613366-32144-0/minicluster-data/ts-1/data,/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786512017613366-32144-0/minicluster-data/ts-1/wal
uuid: "ce209428367c4501b2008b986015ee4f"
format_stamp: "Formatted at 2026-08-12 05:21:15 on dist-test-slave-42z9"
server_key: "22d021ddf760ce60cdca6b111a9ed6f0"
server_key_iv: "407fe8154c028c7e3552616b0c66edb0"
server_key_version: "encryptionkey@0"
I20260812 05:21:15.685485 17016 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786512017613366-32144-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786512017613366-32144-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786512017613366-32144-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 05:21:15.699261 17016 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 05:21:15.700920 17016 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 05:21:15.701050 17016 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 05:21:15.701257 17016 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 05:21:15.701622 17016 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 05:21:15.701660 17016 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 05:21:15.701687 17016 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 05:21:15.701722 17016 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 05:21:15.707213 17016 rpc_server.cc:307] RPC server started. Bound to: 127.31.100.2:43133
I20260812 05:21:15.707267 17236 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.100.2:43133 every 8 connection(s)
I20260812 05:21:15.707540 17016 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786512017613366-32144-0/minicluster-data/ts-1/data/info.pb
Aug 12 05:21:15 dist-test-slave-42z9 krb5kdc[16582](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512075, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.100.2@KRBTEST.COM for kudu/127.31.100.62@KRBTEST.COM
I20260812 05:21:15.713611 32144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0waFvT/build/release/bin/kudu as pid 17016
I20260812 05:21:15.713708 32144 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786512017613366-32144-0/minicluster-data/ts-1/wal/instance
I20260812 05:21:15.713932 32144 external_mini_cluster.cc:1428] Setting key 08fa0bf7dd4ae44ae7e0413b30b4fcda
I20260812 05:21:15.716071 16741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:21:15.708194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.100.2:55315 (local address 127.31.100.62:37923)
0812 05:21:15.708248 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0812 05:21:15.708250 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 05:21:15.708431 (+ 181us) server_negotiation.cc:407] Connection header received
0812 05:21:15.708917 (+ 486us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:21:15.708919 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:21:15.708930 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:21:15.708983 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:21:15.709883 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:21:15.710372 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:21:15.711283 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:21:15.711474 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:21:15.711517 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:21:15.713434 (+ 1917us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 05:21:15.713435 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 05:21:15.713436 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 05:21:15.713466 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0812 05:21:15.714809 (+ 1343us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:21:15.715218 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:21:15.715219 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:21:15.715219 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:21:15.715275 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:21:15.715490 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:21:15.715491 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:21:15.715491 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:21:15.715600 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 05:21:15.715655 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0812 05:21:15.715901 (+ 246us) server_negotiation.cc:299] Negotiation successful
0812 05:21:15.715985 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260812 05:21:15.716243 17240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:21:15.708144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.62:37923 (local address 127.31.100.2:55315)
0812 05:21:15.708305 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0812 05:21:15.708345 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0812 05:21:15.708810 (+ 465us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:21:15.709042 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:21:15.709045 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:21:15.709632 (+ 587us) client_negotiation.cc:189] Negotiated authn=SASL
0812 05:21:15.709811 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:21:15.709816 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:21:15.710461 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:21:15.710462 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:21:15.711224 (+ 762us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:21:15.711226 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:21:15.711524 (+ 298us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:21:15.711525 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:21:15.711615 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:21:15.712047 (+ 432us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 05:21:15.712055 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0812 05:21:15.713315 (+ 1260us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 05:21:15.714894 (+ 1579us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 05:21:15.714900 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 05:21:15.714902 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 05:21:15.715148 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 05:21:15.715329 (+ 181us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 05:21:15.715330 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 05:21:15.715330 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 05:21:15.715434 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 05:21:15.715655 (+ 221us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 05:21:15.715661 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 05:21:15.715787 (+ 126us) client_negotiation.cc:769] Sending connection context
0812 05:21:15.715871 (+ 84us) client_negotiation.cc:240] Negotiation successful
0812 05:21:15.716033 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":76,"threads_started":1}
I20260812 05:21:15.716631 17238 heartbeater.cc:347] Connected to a master server at 127.31.100.62:37923
I20260812 05:21:15.716733 17238 heartbeater.cc:464] Registering TS with master...
I20260812 05:21:15.716905 17238 heartbeater.cc:511] Master 127.31.100.62:37923 requested a full tablet report, sending...
I20260812 05:21:15.717255 16655 ts_manager.cc:194] Registered new tserver with Master: ce209428367c4501b2008b986015ee4f (127.31.100.2:43133)
I20260812 05:21:15.717650 16655 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.100.2@KRBTEST.COM'} at 127.31.100.2:55315
WARNING: no policy specified for kudu/127.31.100.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.100.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.100.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.3.keytab.
Entry for principal kudu/127.31.100.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.3.keytab.
I20260812 05:21:15.737934 32144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0waFvT/build/release/bin/kudu
/tmp/dist-test-task0waFvT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786512017613366-32144-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786512017613366-32144-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786512017613366-32144-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786512017613366-32144-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.100.3:0
--local_ip_for_outbound_sockets=127.31.100.3
--webserver_interface=127.31.100.3
--webserver_port=0
--tserver_master_addrs=127.31.100.62:37923
--keytab_file=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.3.keytab
--principal=kudu/127.31.100.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.100.20:46431
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786512017613366-32144-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786512017613366-32144-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786512017613366-32144-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 05:21:15.817404 17248 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 05:21:15.817716 17248 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 05:21:15.817750 17248 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 05:21:15.819180 17248 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 05:21:15.819236 17248 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 05:21:15.819271 17248 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.100.3
I20260812 05:21:15.821098 17248 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.100.20:46431
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786512017613366-32144-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786512017613366-32144-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.3.keytab
--principal=kudu/127.31.100.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.100.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786512017613366-32144-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.100.3
--webserver_port=0
--tserver_master_addrs=127.31.100.62:37923
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17248
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.100.3
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786512017613366-32144-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 04:20:34 UTC on ccycloud.quasar-phqdqp.root.comops.site
build id 4581871
I20260812 05:21:15.821339 17248 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 05:21:15.821571 17248 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 05:21:15.826315 17260 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 05:21:15.826303 17259 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 05:21:15.826298 17266 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 05:21:15.826510 17248 server_base.cc:1034] running on GCE node
I20260812 05:21:15.826692 17248 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 05:21:15.826911 17248 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 05:21:15.828063 17248 hybrid_clock.cc:648] HybridClock initialized: now 1786512075828050 us; error 31 us; skew 500 ppm
Aug 12 05:21:15 dist-test-slave-42z9 krb5kdc[16582](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512075, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.100.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 05:21:15.830441 17248 init.cc:377] Logged in from keytab as kudu/127.31.100.3@KRBTEST.COM (short username kudu)
I20260812 05:21:15.830857 17248 webserver.cc:460] Webserver started at http://127.31.100.3:39737/ using document root <none> and password file <none>
I20260812 05:21:15.831074 17248 fs_manager.cc:359] Metadata directory not provided
I20260812 05:21:15.831121 17248 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 05:21:15.831243 17248 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 05:21:15.832182 17248 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786512017613366-32144-0/minicluster-data/ts-2/data/instance:
uuid: "349a40d21f784d809ba31eed79c9263f"
format_stamp: "Formatted at 2026-08-12 05:21:15 on dist-test-slave-42z9"
server_key: "d79c447fba001b5391f13f6c11ad3ba2"
server_key_iv: "681dfa3288dc581dd156b9b4549fca02"
server_key_version: "encryptionkey@0"
I20260812 05:21:15.832496 17248 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786512017613366-32144-0/minicluster-data/ts-2/wal/instance:
uuid: "349a40d21f784d809ba31eed79c9263f"
format_stamp: "Formatted at 2026-08-12 05:21:15 on dist-test-slave-42z9"
server_key: "d79c447fba001b5391f13f6c11ad3ba2"
server_key_iv: "681dfa3288dc581dd156b9b4549fca02"
server_key_version: "encryptionkey@0"
I20260812 05:21:15.833739 17248 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260812 05:21:15.834380 17280 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 05:21:15.834529 17248 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260812 05:21:15.834612 17248 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786512017613366-32144-0/minicluster-data/ts-2/data,/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786512017613366-32144-0/minicluster-data/ts-2/wal
uuid: "349a40d21f784d809ba31eed79c9263f"
format_stamp: "Formatted at 2026-08-12 05:21:15 on dist-test-slave-42z9"
server_key: "d79c447fba001b5391f13f6c11ad3ba2"
server_key_iv: "681dfa3288dc581dd156b9b4549fca02"
server_key_version: "encryptionkey@0"
I20260812 05:21:15.834668 17248 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786512017613366-32144-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786512017613366-32144-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786512017613366-32144-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 05:21:15.861166 17248 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 05:21:15.862914 17248 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 05:21:15.863050 17248 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 05:21:15.863251 17248 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 05:21:15.863526 17248 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 05:21:15.863559 17248 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 05:21:15.863586 17248 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 05:21:15.863610 17248 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 05:21:15.869166 17248 rpc_server.cc:307] RPC server started. Bound to: 127.31.100.3:38447
I20260812 05:21:15.869208 17495 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.100.3:38447 every 8 connection(s)
I20260812 05:21:15.869541 17248 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786512017613366-32144-0/minicluster-data/ts-2/data/info.pb
I20260812 05:21:15.872495 32144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0waFvT/build/release/bin/kudu as pid 17248
I20260812 05:21:15.872572 32144 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786512017613366-32144-0/minicluster-data/ts-2/wal/instance
I20260812 05:21:15.872717 32144 external_mini_cluster.cc:1428] Setting key fdb66e55902a3179bbdb15463b871188
Aug 12 05:21:15 dist-test-slave-42z9 krb5kdc[16582](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512075, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.100.3@KRBTEST.COM for kudu/127.31.100.62@KRBTEST.COM
I20260812 05:21:15.878014 16741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:21:15.870184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.100.3:49037 (local address 127.31.100.62:37923)
0812 05:21:15.870240 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0812 05:21:15.870242 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 05:21:15.870417 (+ 175us) server_negotiation.cc:407] Connection header received
0812 05:21:15.870879 (+ 462us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:21:15.870880 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:21:15.870889 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:21:15.870944 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:21:15.871797 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:21:15.872308 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:21:15.873480 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:21:15.873726 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:21:15.873780 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:21:15.875678 (+ 1898us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 05:21:15.875679 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 05:21:15.875679 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 05:21:15.875687 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0812 05:21:15.876722 (+ 1035us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:21:15.877089 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:21:15.877091 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:21:15.877091 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:21:15.877150 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:21:15.877369 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:21:15.877370 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:21:15.877370 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:21:15.877527 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 05:21:15.877585 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0812 05:21:15.877835 (+ 250us) server_negotiation.cc:299] Negotiation successful
0812 05:21:15.877927 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260812 05:21:15.878144 17498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:21:15.870154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.62:37923 (local address 127.31.100.3:49037)
0812 05:21:15.870302 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0812 05:21:15.870341 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0812 05:21:15.870793 (+ 452us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:21:15.871001 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:21:15.871005 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:21:15.871542 (+ 537us) client_negotiation.cc:189] Negotiated authn=SASL
0812 05:21:15.871721 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:21:15.871725 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:21:15.872381 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:21:15.872382 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:21:15.873386 (+ 1004us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:21:15.873388 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:21:15.873801 (+ 413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:21:15.873802 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:21:15.873911 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:21:15.874265 (+ 354us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 05:21:15.874273 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0812 05:21:15.875590 (+ 1317us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 05:21:15.876794 (+ 1204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 05:21:15.876800 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 05:21:15.876802 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 05:21:15.877015 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 05:21:15.877204 (+ 189us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 05:21:15.877205 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 05:21:15.877205 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 05:21:15.877312 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 05:21:15.877584 (+ 272us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 05:21:15.877590 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 05:21:15.877718 (+ 128us) client_negotiation.cc:769] Sending connection context
0812 05:21:15.877804 (+ 86us) client_negotiation.cc:240] Negotiation successful
0812 05:21:15.877947 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":80,"threads_started":1}
I20260812 05:21:15.878566 17496 heartbeater.cc:347] Connected to a master server at 127.31.100.62:37923
I20260812 05:21:15.878650 17496 heartbeater.cc:464] Registering TS with master...
I20260812 05:21:15.878783 17496 heartbeater.cc:511] Master 127.31.100.62:37923 requested a full tablet report, sending...
I20260812 05:21:15.879092 16655 ts_manager.cc:194] Registered new tserver with Master: 349a40d21f784d809ba31eed79c9263f (127.31.100.3:38447)
I20260812 05:21:15.879436 16655 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.100.3@KRBTEST.COM'} at 127.31.100.3:49037
I20260812 05:21:15.884162 32144 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 05:21:15.890621 16741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:21:15.885342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35906 (local address 127.31.100.62:37923)
0812 05:21:15.885396 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0812 05:21:15.885397 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0812 05:21:15.885513 (+ 116us) server_negotiation.cc:407] Connection header received
0812 05:21:15.885581 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:21:15.885581 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:21:15.885590 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:21:15.885639 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:21:15.886145 (+ 506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:21:15.886595 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:21:15.887264 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:21:15.887491 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:21:15.887544 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:21:15.888553 (+ 1009us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 05:21:15.888554 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 05:21:15.888555 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 05:21:15.888561 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0812 05:21:15.889639 (+ 1078us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:21:15.889925 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:21:15.889927 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:21:15.889927 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:21:15.889979 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:21:15.890141 (+ 162us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:21:15.890142 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:21:15.890142 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:21:15.890278 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 05:21:15.890362 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0812 05:21:15.890471 (+ 109us) server_negotiation.cc:299] Negotiation successful
0812 05:21:15.890539 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260812 05:21:15.891947 16655 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35906:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 05:21:15.893527 16655 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 05:21:15.899648 17513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:21:15.895625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.2:43133 (local address 127.0.0.1:51800)
0812 05:21:15.896046 (+ 421us) negotiation.cc:107] Waiting for socket to connect
0812 05:21:15.896055 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0812 05:21:15.896113 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:21:15.896343 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:21:15.896344 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:21:15.896346 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 05:21:15.896385 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:21:15.896386 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:21:15.897742 (+ 1356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:21:15.897743 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:21:15.898563 (+ 820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:21:15.898565 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:21:15.899403 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:21:15.899404 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:21:15.899499 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:21:15.899509 (+ 10us) client_negotiation.cc:769] Sending connection context
0812 05:21:15.899560 (+ 51us) client_negotiation.cc:240] Negotiation successful
0812 05:21:15.899596 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":399,"mutex_wait_us":1,"thread_start_us":81,"threads_started":1}
I20260812 05:21:15.899710 17524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:21:15.895854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51800 (local address 127.31.100.2:43133)
0812 05:21:15.896023 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0812 05:21:15.896026 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 05:21:15.896101 (+ 75us) server_negotiation.cc:407] Connection header received
0812 05:21:15.896176 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:21:15.896178 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:21:15.896192 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:21:15.896261 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 05:21:15.896536 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:21:15.897653 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:21:15.898647 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:21:15.899306 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:21:15.899391 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:21:15.899457 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0812 05:21:15.899587 (+ 130us) server_negotiation.cc:299] Negotiation successful
0812 05:21:15.899629 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":99,"threads_started":1}
I20260812 05:21:15.899950 17512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:21:15.895622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.1:41991 (local address 127.0.0.1:48254)
0812 05:21:15.895928 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0812 05:21:15.895981 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0812 05:21:15.896056 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:21:15.896242 (+ 186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:21:15.896244 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:21:15.896246 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 05:21:15.896320 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:21:15.896322 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:21:15.897638 (+ 1316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:21:15.897639 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:21:15.898808 (+ 1169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:21:15.898812 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:21:15.899716 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:21:15.899717 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:21:15.899784 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:21:15.899788 (+ 4us) client_negotiation.cc:769] Sending connection context
0812 05:21:15.899835 (+ 47us) client_negotiation.cc:240] Negotiation successful
0812 05:21:15.899873 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280,"mutex_wait_us":4,"thread_start_us":89,"threads_started":1}
I20260812 05:21:15.900177 17517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:21:15.895736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48254 (local address 127.31.100.1:41991)
0812 05:21:15.895914 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0812 05:21:15.895919 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 05:21:15.896042 (+ 123us) server_negotiation.cc:407] Connection header received
0812 05:21:15.896135 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:21:15.896138 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:21:15.896154 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:21:15.896222 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 05:21:15.896392 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:21:15.897542 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:21:15.898894 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:21:15.899622 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:21:15.899693 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:21:15.899746 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0812 05:21:15.900052 (+ 306us) server_negotiation.cc:299] Negotiation successful
0812 05:21:15.900099 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":91,"threads_started":1}
I20260812 05:21:15.900718 17516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:21:15.895626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.3:38447 (local address 127.0.0.1:51746)
0812 05:21:15.896158 (+ 532us) negotiation.cc:107] Waiting for socket to connect
0812 05:21:15.896166 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0812 05:21:15.896210 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:21:15.896621 (+ 411us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:21:15.896622 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:21:15.896624 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 05:21:15.896660 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:21:15.896662 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:21:15.897518 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:21:15.897519 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:21:15.898810 (+ 1291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:21:15.898812 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:21:15.900485 (+ 1673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:21:15.900486 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:21:15.900551 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:21:15.900556 (+ 5us) client_negotiation.cc:769] Sending connection context
0812 05:21:15.900613 (+ 57us) client_negotiation.cc:240] Negotiation successful
0812 05:21:15.900652 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":516,"mutex_wait_us":34,"thread_start_us":55,"threads_started":1}
I20260812 05:21:15.900686 17133 tablet_service.cc:1467] Processing CreateTablet for tablet edd16f69092d4534830a5e1a5468be5a (DEFAULT_TABLE table=test-table [id=b6fe9ffdde414ec59a0741685c295086]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 05:21:15.900823 16903 tablet_service.cc:1467] Processing CreateTablet for tablet edd16f69092d4534830a5e1a5468be5a (DEFAULT_TABLE table=test-table [id=b6fe9ffdde414ec59a0741685c295086]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 05:21:15.901041 17522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:21:15.895844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51746 (local address 127.31.100.3:38447)
0812 05:21:15.895986 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0812 05:21:15.895990 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 05:21:15.896397 (+ 407us) server_negotiation.cc:407] Connection header received
0812 05:21:15.896435 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:21:15.896437 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:21:15.896451 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:21:15.896525 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 05:21:15.896731 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:21:15.897407 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:21:15.899783 (+ 2376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:21:15.900413 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:21:15.900483 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:21:15.900536 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0812 05:21:15.900624 (+ 88us) server_negotiation.cc:299] Negotiation successful
0812 05:21:15.900663 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":61,"threads_started":1}
I20260812 05:21:15.901072 17133 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet edd16f69092d4534830a5e1a5468be5a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 05:21:15.901139 16903 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet edd16f69092d4534830a5e1a5468be5a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 05:21:15.901683 17375 tablet_service.cc:1467] Processing CreateTablet for tablet edd16f69092d4534830a5e1a5468be5a (DEFAULT_TABLE table=test-table [id=b6fe9ffdde414ec59a0741685c295086]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 05:21:15.901980 17375 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet edd16f69092d4534830a5e1a5468be5a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 05:21:15.903836 17526 tablet_bootstrap.cc:492] T edd16f69092d4534830a5e1a5468be5a P 949f45ad4d304bde99cf4bc402002bcb: Bootstrap starting.
I20260812 05:21:15.904407 17527 tablet_bootstrap.cc:492] T edd16f69092d4534830a5e1a5468be5a P ce209428367c4501b2008b986015ee4f: Bootstrap starting.
I20260812 05:21:15.904482 17526 tablet_bootstrap.cc:654] T edd16f69092d4534830a5e1a5468be5a P 949f45ad4d304bde99cf4bc402002bcb: Neither blocks nor log segments found. Creating new log.
I20260812 05:21:15.904881 17532 tablet_bootstrap.cc:492] T edd16f69092d4534830a5e1a5468be5a P 349a40d21f784d809ba31eed79c9263f: Bootstrap starting.
I20260812 05:21:15.905059 17527 tablet_bootstrap.cc:654] T edd16f69092d4534830a5e1a5468be5a P ce209428367c4501b2008b986015ee4f: Neither blocks nor log segments found. Creating new log.
I20260812 05:21:15.905189 17526 log.cc:824] T edd16f69092d4534830a5e1a5468be5a P 949f45ad4d304bde99cf4bc402002bcb: Log is configured to *not* fsync() on all Append() calls
I20260812 05:21:15.905475 17532 tablet_bootstrap.cc:654] T edd16f69092d4534830a5e1a5468be5a P 349a40d21f784d809ba31eed79c9263f: Neither blocks nor log segments found. Creating new log.
I20260812 05:21:15.905647 17527 log.cc:824] T edd16f69092d4534830a5e1a5468be5a P ce209428367c4501b2008b986015ee4f: Log is configured to *not* fsync() on all Append() calls
I20260812 05:21:15.906073 17526 tablet_bootstrap.cc:492] T edd16f69092d4534830a5e1a5468be5a P 949f45ad4d304bde99cf4bc402002bcb: No bootstrap required, opened a new log
I20260812 05:21:15.906081 17532 log.cc:824] T edd16f69092d4534830a5e1a5468be5a P 349a40d21f784d809ba31eed79c9263f: Log is configured to *not* fsync() on all Append() calls
I20260812 05:21:15.906149 17526 ts_tablet_manager.cc:1397] T edd16f69092d4534830a5e1a5468be5a P 949f45ad4d304bde99cf4bc402002bcb: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260812 05:21:15.906327 17527 tablet_bootstrap.cc:492] T edd16f69092d4534830a5e1a5468be5a P ce209428367c4501b2008b986015ee4f: No bootstrap required, opened a new log
I20260812 05:21:15.906401 17527 ts_tablet_manager.cc:1397] T edd16f69092d4534830a5e1a5468be5a P ce209428367c4501b2008b986015ee4f: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260812 05:21:15.906802 17532 tablet_bootstrap.cc:492] T edd16f69092d4534830a5e1a5468be5a P 349a40d21f784d809ba31eed79c9263f: No bootstrap required, opened a new log
I20260812 05:21:15.906874 17532 ts_tablet_manager.cc:1397] T edd16f69092d4534830a5e1a5468be5a P 349a40d21f784d809ba31eed79c9263f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260812 05:21:15.907799 17527 raft_consensus.cc:348] T edd16f69092d4534830a5e1a5468be5a P ce209428367c4501b2008b986015ee4f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "949f45ad4d304bde99cf4bc402002bcb" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 41991 } } peers { permanent_uuid: "ce209428367c4501b2008b986015ee4f" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 43133 } } peers { permanent_uuid: "349a40d21f784d809ba31eed79c9263f" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 38447 } }
I20260812 05:21:15.907935 17527 raft_consensus.cc:374] T edd16f69092d4534830a5e1a5468be5a P ce209428367c4501b2008b986015ee4f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 05:21:15.907970 17527 raft_consensus.cc:729] T edd16f69092d4534830a5e1a5468be5a P ce209428367c4501b2008b986015ee4f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce209428367c4501b2008b986015ee4f, State: Initialized, Role: FOLLOWER
I20260812 05:21:15.908104 17527 consensus_queue.cc:260] T edd16f69092d4534830a5e1a5468be5a P ce209428367c4501b2008b986015ee4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "949f45ad4d304bde99cf4bc402002bcb" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 41991 } } peers { permanent_uuid: "ce209428367c4501b2008b986015ee4f" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 43133 } } peers { permanent_uuid: "349a40d21f784d809ba31eed79c9263f" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 38447 } }
I20260812 05:21:15.908056 17526 raft_consensus.cc:348] T edd16f69092d4534830a5e1a5468be5a P 949f45ad4d304bde99cf4bc402002bcb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "949f45ad4d304bde99cf4bc402002bcb" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 41991 } } peers { permanent_uuid: "ce209428367c4501b2008b986015ee4f" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 43133 } } peers { permanent_uuid: "349a40d21f784d809ba31eed79c9263f" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 38447 } }
I20260812 05:21:15.908205 17526 raft_consensus.cc:374] T edd16f69092d4534830a5e1a5468be5a P 949f45ad4d304bde99cf4bc402002bcb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 05:21:15.908506 17527 ts_tablet_manager.cc:1428] T edd16f69092d4534830a5e1a5468be5a P ce209428367c4501b2008b986015ee4f: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260812 05:21:15.908563 17526 raft_consensus.cc:729] T edd16f69092d4534830a5e1a5468be5a P 949f45ad4d304bde99cf4bc402002bcb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 949f45ad4d304bde99cf4bc402002bcb, State: Initialized, Role: FOLLOWER
I20260812 05:21:15.908649 17238 heartbeater.cc:503] Master 127.31.100.62:37923 was elected leader, sending a full tablet report...
I20260812 05:21:15.908744 17526 consensus_queue.cc:260] T edd16f69092d4534830a5e1a5468be5a P 949f45ad4d304bde99cf4bc402002bcb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "949f45ad4d304bde99cf4bc402002bcb" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 41991 } } peers { permanent_uuid: "ce209428367c4501b2008b986015ee4f" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 43133 } } peers { permanent_uuid: "349a40d21f784d809ba31eed79c9263f" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 38447 } }
I20260812 05:21:15.908779 17532 raft_consensus.cc:348] T edd16f69092d4534830a5e1a5468be5a P 349a40d21f784d809ba31eed79c9263f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "949f45ad4d304bde99cf4bc402002bcb" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 41991 } } peers { permanent_uuid: "ce209428367c4501b2008b986015ee4f" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 43133 } } peers { permanent_uuid: "349a40d21f784d809ba31eed79c9263f" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 38447 } }
I20260812 05:21:15.908908 17532 raft_consensus.cc:374] T edd16f69092d4534830a5e1a5468be5a P 349a40d21f784d809ba31eed79c9263f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 05:21:15.908941 17532 raft_consensus.cc:729] T edd16f69092d4534830a5e1a5468be5a P 349a40d21f784d809ba31eed79c9263f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 349a40d21f784d809ba31eed79c9263f, State: Initialized, Role: FOLLOWER
I20260812 05:21:15.909027 17526 ts_tablet_manager.cc:1428] T edd16f69092d4534830a5e1a5468be5a P 949f45ad4d304bde99cf4bc402002bcb: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260812 05:21:15.909039 17532 consensus_queue.cc:260] T edd16f69092d4534830a5e1a5468be5a P 349a40d21f784d809ba31eed79c9263f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "949f45ad4d304bde99cf4bc402002bcb" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 41991 } } peers { permanent_uuid: "ce209428367c4501b2008b986015ee4f" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 43133 } } peers { permanent_uuid: "349a40d21f784d809ba31eed79c9263f" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 38447 } }
I20260812 05:21:15.909144 17009 heartbeater.cc:503] Master 127.31.100.62:37923 was elected leader, sending a full tablet report...
I20260812 05:21:15.909291 17532 ts_tablet_manager.cc:1428] T edd16f69092d4534830a5e1a5468be5a P 349a40d21f784d809ba31eed79c9263f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260812 05:21:15.909518 17496 heartbeater.cc:503] Master 127.31.100.62:37923 was elected leader, sending a full tablet report...
W20260812 05:21:15.958034 17239 tablet.cc:2369] T edd16f69092d4534830a5e1a5468be5a P ce209428367c4501b2008b986015ee4f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 05:21:16.037708 17540 raft_consensus.cc:482] T edd16f69092d4534830a5e1a5468be5a P 949f45ad4d304bde99cf4bc402002bcb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 05:21:16.037858 17540 raft_consensus.cc:504] T edd16f69092d4534830a5e1a5468be5a P 949f45ad4d304bde99cf4bc402002bcb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "949f45ad4d304bde99cf4bc402002bcb" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 41991 } } peers { permanent_uuid: "ce209428367c4501b2008b986015ee4f" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 43133 } } peers { permanent_uuid: "349a40d21f784d809ba31eed79c9263f" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 38447 } }
I20260812 05:21:16.038311 17540 leader_election.cc:288] T edd16f69092d4534830a5e1a5468be5a P 949f45ad4d304bde99cf4bc402002bcb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ce209428367c4501b2008b986015ee4f (127.31.100.2:43133), 349a40d21f784d809ba31eed79c9263f (127.31.100.3:38447)
I20260812 05:21:16.041574 17524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:21:16.038549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.100.1:57123 (local address 127.31.100.2:43133)
0812 05:21:16.038595 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0812 05:21:16.038597 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 05:21:16.038624 (+ 27us) server_negotiation.cc:407] Connection header received
0812 05:21:16.038655 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:21:16.038656 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:21:16.038665 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:21:16.038719 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 05:21:16.038887 (+ 168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:21:16.039674 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:21:16.040642 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:21:16.041204 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:21:16.041265 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:21:16.041274 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0812 05:21:16.041467 (+ 193us) server_negotiation.cc:299] Negotiation successful
0812 05:21:16.041508 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260812 05:21:16.041590 17522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:21:16.038628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.100.1:39505 (local address 127.31.100.3:38447)
0812 05:21:16.038681 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0812 05:21:16.038683 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 05:21:16.038691 (+ 8us) server_negotiation.cc:407] Connection header received
0812 05:21:16.038747 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:21:16.038748 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:21:16.038756 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:21:16.038805 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 05:21:16.038936 (+ 131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:21:16.039678 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:21:16.040620 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:21:16.041189 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:21:16.041265 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:21:16.041274 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0812 05:21:16.041504 (+ 230us) server_negotiation.cc:299] Negotiation successful
0812 05:21:16.041538 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260812 05:21:16.041596 17011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:21:16.038467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.2:43133 (local address 127.31.100.1:57123)
0812 05:21:16.038517 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0812 05:21:16.038529 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0812 05:21:16.038577 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:21:16.038738 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:21:16.038739 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:21:16.038741 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 05:21:16.038815 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:21:16.038817 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:21:16.039746 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:21:16.039747 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:21:16.040529 (+ 782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:21:16.040532 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:21:16.041279 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:21:16.041280 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:21:16.041376 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:21:16.041378 (+ 2us) client_negotiation.cc:769] Sending connection context
0812 05:21:16.041470 (+ 92us) client_negotiation.cc:240] Negotiation successful
0812 05:21:16.041514 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28,"mutex_wait_us":25}
I20260812 05:21:16.041603 17543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:21:16.038525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.3:38447 (local address 127.31.100.1:39505)
0812 05:21:16.038651 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0812 05:21:16.038659 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0812 05:21:16.038695 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:21:16.038812 (+ 117us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:21:16.038812 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:21:16.038814 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 05:21:16.038873 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:21:16.038874 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:21:16.039788 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:21:16.039789 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:21:16.040532 (+ 743us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:21:16.040534 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:21:16.041279 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:21:16.041280 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:21:16.041372 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:21:16.041377 (+ 5us) client_negotiation.cc:769] Sending connection context
0812 05:21:16.041494 (+ 117us) client_negotiation.cc:240] Negotiation successful
0812 05:21:16.041524 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"mutex_wait_us":18,"thread_start_us":77,"threads_started":1}
I20260812 05:21:16.041951 17168 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edd16f69092d4534830a5e1a5468be5a" candidate_uuid: "949f45ad4d304bde99cf4bc402002bcb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce209428367c4501b2008b986015ee4f" is_pre_election: true
I20260812 05:21:16.042115 17168 raft_consensus.cc:2454] T edd16f69092d4534830a5e1a5468be5a P ce209428367c4501b2008b986015ee4f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 949f45ad4d304bde99cf4bc402002bcb in term 0.
I20260812 05:21:16.042136 17406 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edd16f69092d4534830a5e1a5468be5a" candidate_uuid: "949f45ad4d304bde99cf4bc402002bcb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "349a40d21f784d809ba31eed79c9263f" is_pre_election: true
I20260812 05:21:16.042269 17406 raft_consensus.cc:2454] T edd16f69092d4534830a5e1a5468be5a P 349a40d21f784d809ba31eed79c9263f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 949f45ad4d304bde99cf4bc402002bcb in term 0.
I20260812 05:21:16.042357 16812 leader_election.cc:302] T edd16f69092d4534830a5e1a5468be5a P 949f45ad4d304bde99cf4bc402002bcb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 949f45ad4d304bde99cf4bc402002bcb, ce209428367c4501b2008b986015ee4f; no voters:
I20260812 05:21:16.042491 17540 raft_consensus.cc:2781] T edd16f69092d4534830a5e1a5468be5a P 949f45ad4d304bde99cf4bc402002bcb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 05:21:16.042563 17540 raft_consensus.cc:482] T edd16f69092d4534830a5e1a5468be5a P 949f45ad4d304bde99cf4bc402002bcb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 05:21:16.042587 17540 raft_consensus.cc:3037] T edd16f69092d4534830a5e1a5468be5a P 949f45ad4d304bde99cf4bc402002bcb [term 0 FOLLOWER]: Advancing to term 1
I20260812 05:21:16.043320 17540 raft_consensus.cc:504] T edd16f69092d4534830a5e1a5468be5a P 949f45ad4d304bde99cf4bc402002bcb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "949f45ad4d304bde99cf4bc402002bcb" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 41991 } } peers { permanent_uuid: "ce209428367c4501b2008b986015ee4f" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 43133 } } peers { permanent_uuid: "349a40d21f784d809ba31eed79c9263f" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 38447 } }
I20260812 05:21:16.043440 17540 leader_election.cc:288] T edd16f69092d4534830a5e1a5468be5a P 949f45ad4d304bde99cf4bc402002bcb [CANDIDATE]: Term 1 election: Requested vote from peers ce209428367c4501b2008b986015ee4f (127.31.100.2:43133), 349a40d21f784d809ba31eed79c9263f (127.31.100.3:38447)
I20260812 05:21:16.043591 17406 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edd16f69092d4534830a5e1a5468be5a" candidate_uuid: "949f45ad4d304bde99cf4bc402002bcb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "349a40d21f784d809ba31eed79c9263f"
I20260812 05:21:16.043658 17406 raft_consensus.cc:3037] T edd16f69092d4534830a5e1a5468be5a P 349a40d21f784d809ba31eed79c9263f [term 0 FOLLOWER]: Advancing to term 1
I20260812 05:21:16.044301 17406 raft_consensus.cc:2454] T edd16f69092d4534830a5e1a5468be5a P 349a40d21f784d809ba31eed79c9263f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 949f45ad4d304bde99cf4bc402002bcb in term 1.
I20260812 05:21:16.044391 17168 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edd16f69092d4534830a5e1a5468be5a" candidate_uuid: "949f45ad4d304bde99cf4bc402002bcb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce209428367c4501b2008b986015ee4f"
I20260812 05:21:16.044451 16814 leader_election.cc:302] T edd16f69092d4534830a5e1a5468be5a P 949f45ad4d304bde99cf4bc402002bcb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 349a40d21f784d809ba31eed79c9263f, 949f45ad4d304bde99cf4bc402002bcb; no voters:
I20260812 05:21:16.044453 17168 raft_consensus.cc:3037] T edd16f69092d4534830a5e1a5468be5a P ce209428367c4501b2008b986015ee4f [term 0 FOLLOWER]: Advancing to term 1
I20260812 05:21:16.044550 17540 raft_consensus.cc:2781] T edd16f69092d4534830a5e1a5468be5a P 949f45ad4d304bde99cf4bc402002bcb [term 1 FOLLOWER]: Leader election won for term 1
I20260812 05:21:16.044714 17540 raft_consensus.cc:686] T edd16f69092d4534830a5e1a5468be5a P 949f45ad4d304bde99cf4bc402002bcb [term 1 LEADER]: Becoming Leader. State: Replica: 949f45ad4d304bde99cf4bc402002bcb, State: Running, Role: LEADER
I20260812 05:21:16.044812 17540 consensus_queue.cc:237] T edd16f69092d4534830a5e1a5468be5a P 949f45ad4d304bde99cf4bc402002bcb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "949f45ad4d304bde99cf4bc402002bcb" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 41991 } } peers { permanent_uuid: "ce209428367c4501b2008b986015ee4f" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 43133 } } peers { permanent_uuid: "349a40d21f784d809ba31eed79c9263f" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 38447 } }
I20260812 05:21:16.045333 17168 raft_consensus.cc:2454] T edd16f69092d4534830a5e1a5468be5a P ce209428367c4501b2008b986015ee4f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 949f45ad4d304bde99cf4bc402002bcb in term 1.
I20260812 05:21:16.045867 16654 catalog_manager.cc:5495] T edd16f69092d4534830a5e1a5468be5a P 949f45ad4d304bde99cf4bc402002bcb reported cstate change: term changed from 0 to 1, leader changed from <none> to 949f45ad4d304bde99cf4bc402002bcb (127.31.100.1). New cstate: current_term: 1 leader_uuid: "949f45ad4d304bde99cf4bc402002bcb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "949f45ad4d304bde99cf4bc402002bcb" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 41991 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ce209428367c4501b2008b986015ee4f" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 43133 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "349a40d21f784d809ba31eed79c9263f" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 38447 } health_report { overall_health: UNKNOWN } } }
W20260812 05:21:16.055368 17010 tablet.cc:2369] T edd16f69092d4534830a5e1a5468be5a P 949f45ad4d304bde99cf4bc402002bcb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 05:21:16.079656 17517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:21:16.077623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48256 (local address 127.31.100.1:41991)
0812 05:21:16.077674 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0812 05:21:16.077676 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 05:21:16.077684 (+ 8us) server_negotiation.cc:407] Connection header received
0812 05:21:16.077733 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:21:16.077734 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:21:16.077751 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:21:16.077804 (+ 53us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 05:21:16.077908 (+ 104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:21:16.078370 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:21:16.079073 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:21:16.079248 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:21:16.079289 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:21:16.079400 (+ 111us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 05:21:16.079494 (+ 94us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 05:21:16.079548 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0812 05:21:16.079563 (+ 15us) server_negotiation.cc:299] Negotiation successful
0812 05:21:16.079580 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260812 05:21:16.081463 17168 raft_consensus.cc:1261] T edd16f69092d4534830a5e1a5468be5a P ce209428367c4501b2008b986015ee4f [term 1 FOLLOWER]: Refusing update from remote peer 949f45ad4d304bde99cf4bc402002bcb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 05:21:16.081463 17406 raft_consensus.cc:1261] T edd16f69092d4534830a5e1a5468be5a P 349a40d21f784d809ba31eed79c9263f [term 1 FOLLOWER]: Refusing update from remote peer 949f45ad4d304bde99cf4bc402002bcb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 05:21:16.081840 17544 consensus_queue.cc:1035] T edd16f69092d4534830a5e1a5468be5a P 949f45ad4d304bde99cf4bc402002bcb [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce209428367c4501b2008b986015ee4f" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 43133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 05:21:16.081972 17540 consensus_queue.cc:1035] T edd16f69092d4534830a5e1a5468be5a P 949f45ad4d304bde99cf4bc402002bcb [LEADER]: Connected to new peer: Peer: permanent_uuid: "349a40d21f784d809ba31eed79c9263f" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 38447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 05:21:16.084797 17548 mvcc.cc:205] Tried to move back new op lower bound from 7317553463626776576 to 7317553463480213504. Current Snapshot: MvccSnapshot[applied={T|T < 7317553463626776576}]
I20260812 05:21:16.084892 17551 mvcc.cc:205] Tried to move back new op lower bound from 7317553463626776576 to 7317553463480213504. Current Snapshot: MvccSnapshot[applied={T|T < 7317553463626776576}]
I20260812 05:21:16.087045 16655 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35906:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 05:21:16.087129 16655 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35906:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 05:21:16.087816 16655 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fd0e2476c9b1458fb2a281641ae24e52: Sending DeleteTablet for 3 replicas of tablet edd16f69092d4534830a5e1a5468be5a
I20260812 05:21:16.088161 17133 tablet_service.cc:1514] Processing DeleteTablet for tablet edd16f69092d4534830a5e1a5468be5a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 05:21:16 UTC) from {username='kudu', principal='kudu/127.31.100.62@KRBTEST.COM'} at 127.0.0.1:51800
I20260812 05:21:16.088267 17375 tablet_service.cc:1514] Processing DeleteTablet for tablet edd16f69092d4534830a5e1a5468be5a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 05:21:16 UTC) from {username='kudu', principal='kudu/127.31.100.62@KRBTEST.COM'} at 127.0.0.1:51746
I20260812 05:21:16.088382 17564 tablet_replica.cc:330] T edd16f69092d4534830a5e1a5468be5a P ce209428367c4501b2008b986015ee4f: stopping tablet replica
I20260812 05:21:16.088399 16903 tablet_service.cc:1514] Processing DeleteTablet for tablet edd16f69092d4534830a5e1a5468be5a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 05:21:16 UTC) from {username='kudu', principal='kudu/127.31.100.62@KRBTEST.COM'} at 127.0.0.1:48254
I20260812 05:21:16.088454 17565 tablet_replica.cc:330] T edd16f69092d4534830a5e1a5468be5a P 349a40d21f784d809ba31eed79c9263f: stopping tablet replica
I20260812 05:21:16.088479 17564 raft_consensus.cc:2229] T edd16f69092d4534830a5e1a5468be5a P ce209428367c4501b2008b986015ee4f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 05:21:16.088536 17565 raft_consensus.cc:2229] T edd16f69092d4534830a5e1a5468be5a P 349a40d21f784d809ba31eed79c9263f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 05:21:16.088610 17564 raft_consensus.cc:2258] T edd16f69092d4534830a5e1a5468be5a P ce209428367c4501b2008b986015ee4f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 05:21:16.088624 17565 raft_consensus.cc:2258] T edd16f69092d4534830a5e1a5468be5a P 349a40d21f784d809ba31eed79c9263f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 05:21:16.088702 32144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0waFvT/build/release/bin/kudu with pid 16775
I20260812 05:21:16.095391 17564 ts_tablet_manager.cc:1905] T edd16f69092d4534830a5e1a5468be5a P ce209428367c4501b2008b986015ee4f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 05:21:16.096975 17564 ts_tablet_manager.cc:1918] T edd16f69092d4534830a5e1a5468be5a P ce209428367c4501b2008b986015ee4f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 05:21:16.097045 17564 log.cc:1196] T edd16f69092d4534830a5e1a5468be5a P ce209428367c4501b2008b986015ee4f: Deleting WAL directory at /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786512017613366-32144-0/minicluster-data/ts-1/wal/wals/edd16f69092d4534830a5e1a5468be5a
I20260812 05:21:16.097281 17564 ts_tablet_manager.cc:1939] T edd16f69092d4534830a5e1a5468be5a P ce209428367c4501b2008b986015ee4f: Deleting consensus metadata
I20260812 05:21:16.097301 17565 ts_tablet_manager.cc:1905] T edd16f69092d4534830a5e1a5468be5a P 349a40d21f784d809ba31eed79c9263f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 05:21:16.098532 16630 catalog_manager.cc:4841] TS ce209428367c4501b2008b986015ee4f (127.31.100.2:43133): tablet edd16f69092d4534830a5e1a5468be5a (table test-table [id=b6fe9ffdde414ec59a0741685c295086]) successfully deleted
I20260812 05:21:16.099606 17565 ts_tablet_manager.cc:1918] T edd16f69092d4534830a5e1a5468be5a P 349a40d21f784d809ba31eed79c9263f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 05:21:16.099687 17565 log.cc:1196] T edd16f69092d4534830a5e1a5468be5a P 349a40d21f784d809ba31eed79c9263f: Deleting WAL directory at /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786512017613366-32144-0/minicluster-data/ts-2/wal/wals/edd16f69092d4534830a5e1a5468be5a
I20260812 05:21:16.100088 17565 ts_tablet_manager.cc:1939] T edd16f69092d4534830a5e1a5468be5a P 349a40d21f784d809ba31eed79c9263f: Deleting consensus metadata
I20260812 05:21:16.100999 16633 catalog_manager.cc:4841] TS 349a40d21f784d809ba31eed79c9263f (127.31.100.3:38447): tablet edd16f69092d4534830a5e1a5468be5a (table test-table [id=b6fe9ffdde414ec59a0741685c295086]) successfully deleted
W20260812 05:21:16.107709 16631 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.31.100.1:41991 (error 108)
I20260812 05:21:16.107998 32144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0waFvT/build/release/bin/kudu with pid 17016
I20260812 05:21:16.108219 17516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:21:16.108109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.1:41991 (local address 127.0.0.1:48266)
0812 05:21:16.108155 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0812 05:21:16.108188 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.100.1:41991: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260812 05:21:16.108408 16631 catalog_manager.cc:4584] TS 949f45ad4d304bde99cf4bc402002bcb (127.31.100.1:41991): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet edd16f69092d4534830a5e1a5468be5a: Network error: Client connection negotiation failed: client connection to 127.31.100.1:41991: connect: Connection refused (error 111)
I20260812 05:21:16.118268 32144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0waFvT/build/release/bin/kudu with pid 17248
I20260812 05:21:16.161824 17516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:21:16.161610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.1:41991 (local address 127.0.0.1:48280)
0812 05:21:16.161699 (+ 89us) negotiation.cc:107] Waiting for socket to connect
0812 05:21:16.161761 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.100.1:41991: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260812 05:21:16.162583 17516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:21:16.162359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.1:41991 (local address 127.0.0.1:48288)
0812 05:21:16.162447 (+ 88us) negotiation.cc:107] Waiting for socket to connect
0812 05:21:16.162496 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.100.1:41991: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260812 05:21:16.167768 32144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0waFvT/build/release/bin/kudu with pid 16605
2026-08-12T05:21:16Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2244 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786512017613366-32144-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 05:21:16 dist-test-slave-42z9 krb5kdc[17576](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 05:21:16 dist-test-slave-42z9 krb5kdc[17576](info): set up 2 sockets
Aug 12 05:21:16 dist-test-slave-42z9 krb5kdc[17576](info): commencing operation
krb5kdc: starting...
W20260812 05:21:17.524367 32144 mini_kdc.cc:121] Time spent starting KDC: real 1.320s user 0.000s sys 0.010s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786512017613366-32144-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786512017613366-32144-0/krb5kdc/test-user.keytab.
Aug 12 05:21:17 dist-test-slave-42z9 krb5kdc[17576](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512077, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T05:21:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T05:21:17Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.31.100.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.31.100.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.31.100.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786512017613366-32144-0/krb5kdc/rangeradmin_127.31.100.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.31.100.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786512017613366-32144-0/krb5kdc/rangeradmin_127.31.100.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.31.100.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.31.100.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.31.100.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786512017613366-32144-0/krb5kdc/rangerlookup_127.31.100.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.31.100.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786512017613366-32144-0/krb5kdc/rangerlookup_127.31.100.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.31.100.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.31.100.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.31.100.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786512017613366-32144-0/krb5kdc/HTTP_127.31.100.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.31.100.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786512017613366-32144-0/krb5kdc/HTTP_127.31.100.20@KRBTEST.COM.keytab.
I20260812 05:21:17.677215 32144 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786512017613366-32144-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-task0waFvT/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786512017613366-32144-0/minicluster-data/postgres -l logfile start
2026-08-12 05:21:20.036 UTC [17627] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-08-12 05:21:20.036 UTC [17627] LOG: listening on IPv4 address "127.31.100.20", port 40333
2026-08-12 05:21:20.040 UTC [17627] LOG: listening on Unix socket "/tmp/.s.PGSQL.40333"
2026-08-12 05:21:20.068 UTC [17636] LOG: database system was shut down at 2026-08-12 05:21:18 UTC
2026-08-12 05:21:20.072 UTC [17627] LOG: database system is ready to accept connections
I20260812 05:21:21.140934 32144 mini_postgres.cc:96] Postgres bound to 40333
2026-08-12 05:21:21.149 UTC [17651] FATAL: database "slave" does not exist
127.31.100.20:40333 - accepting connections
I20260812 05:21:21.150084 32144 mini_ranger.cc:161] Starting Ranger...
I20260812 05:21:21.169212 32144 mini_ranger.cc:84] Created miniranger Postgres user
I20260812 05:21:21.610143 32144 mini_ranger.cc:87] Created ranger Postgres database
I20260812 05:21:21.610227 32144 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786512017613366-32144-0/minicluster-data/ranger-admin
2026-08-12 05:21:21,940 [I] DB FLAVOR :POSTGRES
2026-08-12 05:21:21,941 [I] --------- Verifying Ranger DB connection ---------
2026-08-12 05:21:21,941 [I] Checking connection..
2026-08-12 05:21:21,941 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0waFvT/build/release/bin/postgresql.jar:/tmp/dist-test-task0waFvT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.100.20:40333/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-08-12 05:21:22,207 [I] Checking connection passed.
2026-08-12 05:21:22,207 [I] --------- Verifying version history table ---------
2026-08-12 05:21:22,207 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0waFvT/build/release/bin/postgresql.jar:/tmp/dist-test-task0waFvT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.100.20:40333/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-12 05:21:22,466 [I] Table x_db_version_h does not exist in database ranger
2026-08-12 05:21:22,466 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0waFvT/build/release/bin/postgresql.jar:/tmp/dist-test-task0waFvT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.100.20:40333/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-12 05:21:22,722 [I] Table x_db_version_h does not exist in database ranger
2026-08-12 05:21:22,722 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-08-12 05:21:22,722 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0waFvT/build/release/bin/postgresql.jar:/tmp/dist-test-task0waFvT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.100.20:40333/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task0waFvT/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-08-12 05:21:22.960 UTC [17759] WARNING: there is no transaction in progress
2026-08-12 05:21:22,969 [I] create_dbversion_catalog.sql file imported successfully
2026-08-12 05:21:22,969 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0waFvT/build/release/bin/postgresql.jar:/tmp/dist-test-task0waFvT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.100.20:40333/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-12 05:21:23,228 [I] Table x_db_version_h already exists in database 'ranger'
2026-08-12 05:21:23,228 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-12 05:21:23,228 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0waFvT/build/release/bin/postgresql.jar:/tmp/dist-test-task0waFvT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.100.20:40333/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-08-12 05:21:23,469 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0waFvT/build/release/bin/postgresql.jar:/tmp/dist-test-task0waFvT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.100.20:40333/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-08-12 05:21:23,721 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0waFvT/build/release/bin/postgresql.jar:/tmp/dist-test-task0waFvT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.100.20:40333/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-42z9.c.gcp-upstream.internal','N') ;"
2026-08-12 05:21:23,968 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0waFvT/build/release/bin/postgresql.jar:/tmp/dist-test-task0waFvT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.100.20:40333/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-12 05:21:24,225 [I] Table x_portal_user does not exist in database ranger
2026-08-12 05:21:24,225 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0waFvT/build/release/bin/postgresql.jar:/tmp/dist-test-task0waFvT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.100.20:40333/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-12 05:21:24,480 [I] Table x_policy_ref_group does not exist in database ranger
2026-08-12 05:21:24,480 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0waFvT/build/release/bin/postgresql.jar:/tmp/dist-test-task0waFvT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.100.20:40333/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-12 05:21:24,720 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0waFvT/build/release/bin/postgresql.jar:/tmp/dist-test-task0waFvT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.100.20:40333/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-12 05:21:24,968 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-08-12 05:21:24,969 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0waFvT/build/release/bin/postgresql.jar:/tmp/dist-test-task0waFvT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.100.20:40333/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task0waFvT/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-08-12 05:21:25.890 UTC [17967] WARNING: there is no transaction in progress
2026-08-12 05:21:25.906 UTC [17967] WARNING: there is no transaction in progress
2026-08-12 05:21:25.921 UTC [17967] WARNING: there is no transaction in progress
2026-08-12 05:21:25.936 UTC [17967] WARNING: there is no transaction in progress
2026-08-12 05:21:25.951 UTC [17967] WARNING: there is no transaction in progress
2026-08-12 05:21:25.969 UTC [17967] WARNING: there is no transaction in progress
2026-08-12 05:21:26.021 UTC [17967] WARNING: there is no transaction in progress
2026-08-12 05:21:26.027 UTC [17967] WARNING: there is no transaction in progress
2026-08-12 05:21:26.041 UTC [17967] WARNING: there is no transaction in progress
2026-08-12 05:21:26.050 UTC [17967] WARNING: there is no transaction in progress
2026-08-12 05:21:26.060 UTC [17967] WARNING: there is no transaction in progress
2026-08-12 05:21:26.073 UTC [17967] WARNING: there is no transaction in progress
2026-08-12 05:21:26.082 UTC [17967] WARNING: there is no transaction in progress
2026-08-12 05:21:26.088 UTC [17967] WARNING: there is no transaction in progress
2026-08-12 05:21:26.094 UTC [17967] WARNING: there is no transaction in progress
2026-08-12 05:21:26.100 UTC [17967] WARNING: there is no transaction in progress
2026-08-12 05:21:26,674 [I] ranger_core_db_postgres.sql file imported successfully
2026-08-12 05:21:26,674 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0waFvT/build/release/bin/postgresql.jar:/tmp/dist-test-task0waFvT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.100.20:40333/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-08-12 05:21:26,919 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-08-12 05:21:26,919 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0waFvT/build/release/bin/postgresql.jar:/tmp/dist-test-task0waFvT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.100.20:40333/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-12 05:21:27,180 [I] Table x_portal_user already exists in database 'ranger'
2026-08-12 05:21:27,181 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0waFvT/build/release/bin/postgresql.jar:/tmp/dist-test-task0waFvT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.100.20:40333/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-12 05:21:27,436 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-08-12 05:21:27,436 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0waFvT/build/release/bin/postgresql.jar:/tmp/dist-test-task0waFvT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.100.20:40333/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-12 05:21:27,688 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0waFvT/build/release/bin/postgresql.jar:/tmp/dist-test-task0waFvT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.100.20:40333/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-12 05:21:27,941 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0waFvT/build/release/bin/postgresql.jar:/tmp/dist-test-task0waFvT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.100.20:40333/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-42z9.c.gcp-upstream.internal';"
2026-08-12 05:21:28,190 [I] CORE_DB_SCHEMA import status has been updated
2026-08-12 05:21:28,190 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0waFvT/build/release/bin/postgresql.jar:/tmp/dist-test-task0waFvT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.100.20:40333/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-08-12 05:21:28,438 [I] DB_PATCHES have already been applied
I20260812 05:21:28.443964 32144 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786512017613366-32144-0/minicluster-data/ranger-admin:/tmp/dist-test-task0waFvT/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-task0waFvT/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-task0waFvT/build/release/bin/postgresql.jar:/tmp/dist-test-task0waFvT/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260812 05:21:28.444028 32144 mini_ranger.cc:193] Using host: 127.31.100.20
I20260812 05:21:28.447642 32144 mini_ranger.cc:227] Ranger admin URL: http://127.31.100.20:54405
Aug 12, 2026 5:21:28 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 12, 2026 5:21:28 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 12, 2026 5:21:28 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786512017613366-32144-0/minicluster-data/ranger-admin/ews/webapp
Aug 12, 2026 5:21:28 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786512017613366-32144-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Aug 12, 2026 5:21:28 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786512017613366-32144-0/minicluster-data/ranger-admin/ews/webapp] .....
Aug 12, 2026 5:21:29 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786512017613366-32144-0/minicluster-data/ranger-admin/ews/webapp].
Aug 12, 2026 5:21:29 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260812 05:21:29.719156 32144 mini_ranger.cc:160] Time spent starting Ranger: real 8.569s user 0.002s sys 0.008s
Aug 12, 2026 5:21:49 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Aug 12, 2026 5:21:49 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.MetricsREST
Aug 12, 2026 5:21:49 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Aug 12, 2026 5:21:49 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Aug 12, 2026 5:21:49 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Aug 12, 2026 5:21:49 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Aug 12, 2026 5:21:49 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Aug 12, 2026 5:21:49 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Aug 12, 2026 5:21:49 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Aug 12, 2026 5:21:49 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Aug 12, 2026 5:21:49 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Aug 12, 2026 5:21:49 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Aug 12, 2026 5:21:49 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Aug 12, 2026 5:21:49 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Aug 12, 2026 5:21:49 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Aug 12, 2026 5:21:49 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Aug 12, 2026 5:21:49 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Aug 12, 2026 5:21:50 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260812 05:21:50.697573 32144 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.31.100.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.31.100.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.31.100.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786512017613366-32144-0/krb5kdc/rangerkms_127.31.100.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.31.100.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786512017613366-32144-0/krb5kdc/rangerkms_127.31.100.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.31.100.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.31.100.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.31.100.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786512017613366-32144-0/krb5kdc/HTTP_127.31.100.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.31.100.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786512017613366-32144-0/krb5kdc/HTTP_127.31.100.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Aug 12 05:21:50 dist-test-slave-42z9 krb5kdc[17576](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512110, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260812 05:21:50.770907 32144 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260812 05:21:50.780294 32144 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260812 05:21:51.524508 32144 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260812 05:21:51.524585 32144 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786512017613366-32144-0/minicluster-data/ranger-kms
I20260812 05:21:51.524601 32144 mini_ranger_kms.cc:227] Using postgres at 127.31.100.20:40333
2026-08-12 05:21:52,076 [I] DB FLAVOR :POSTGRES
2026-08-12 05:21:52,076 [I] --------- Verifying Ranger DB connection ---------
2026-08-12 05:21:52,076 [I] Checking connection
2026-08-12 05:21:52,077 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0waFvT/build/release/bin/postgresql.jar:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786512017613366-32144-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.100.20:40333/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-08-12 05:21:52,353 [I] connection success
2026-08-12 05:21:52,353 [I] --------- Verifying Ranger DB tables ---------
2026-08-12 05:21:52,353 [I] Verifying table ranger_masterkey in database rangerkms
2026-08-12 05:21:52,353 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0waFvT/build/release/bin/postgresql.jar:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786512017613366-32144-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.100.20:40333/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-08-12 05:21:52,586 [I] Table ranger_masterkey does not exist in database rangerkms
2026-08-12 05:21:52,586 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-12 05:21:52,586 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-08-12 05:21:52,586 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0waFvT/build/release/bin/postgresql.jar:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786512017613366-32144-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.100.20:40333/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786512017613366-32144-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-08-12 05:21:52,834 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260812 05:21:53.314270 32144 mini_ranger_kms.cc:310] Created kms service
I20260812 05:21:53.511126 32144 mini_ranger_kms.cc:326] Created kudu user
I20260812 05:21:53.552302 32144 mini_ranger_kms.cc:343] Created rangerkms user
Aug 12 05:21:53 dist-test-slave-42z9 krb5kdc[17576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512110, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.31.100.20@KRBTEST.COM
I20260812 05:21:53.931279 32144 mini_ranger_kms.cc:384] Added ranger policy
I20260812 05:21:53.931401 32144 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786512017613366-32144-0/minicluster-data/ranger-kms:/tmp/dist-test-task0waFvT/build/release/bin/postgresql.jar:/tmp/dist-test-task0waFvT/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-task0waFvT/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260812 05:21:53.931423 32144 mini_ranger_kms.cc:240] Using host: 127.31.100.20
I20260812 05:21:53.935451 32144 mini_ranger_kms.cc:276] Ranger KMS PID: 18290
I20260812 05:21:53.935575 32144 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.31.100.20:41715
Aug 12, 2026 5:21:55 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 12, 2026 5:21:55 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 12, 2026 5:21:55 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786512017613366-32144-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Aug 12, 2026 5:21:55 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786512017613366-32144-0/minicluster-data/ranger-kms/ews/webapp] .....
Aug 12, 2026 5:21:55 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786512017613366-32144-0/minicluster-data/ranger-kms/ews/webapp].
Aug 12, 2026 5:21:55 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Aug 12 05:21:56 dist-test-slave-42z9 krb5kdc[17576](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786512116, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.31.100.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 05:21:56.414546 32144 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.644s user 0.000s sys 0.450s
I20260812 05:21:56.414657 32144 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260812 05:21:56.414744 32144 mini_ranger_kms.cc:401] 127.31.100.20:41715/kms/v1/keys
Aug 12 05:21:56 dist-test-slave-42z9 krb5kdc[17576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512116, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.31.100.20@KRBTEST.COM for HTTP/127.31.100.20@KRBTEST.COM
Aug 12 05:21:57 dist-test-slave-42z9 krb5kdc[17576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512116, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.31.100.20@KRBTEST.COM for HTTP/127.31.100.20@KRBTEST.COM
Aug 12 05:21:59 dist-test-slave-42z9 krb5kdc[17576](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512119, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.31.100.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.100.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.100.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.62.keytab.
Entry for principal kudu/127.31.100.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.62.keytab.
I20260812 05:21:59.479645 32144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0waFvT/build/release/bin/kudu
/tmp/dist-test-task0waFvT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786512017613366-32144-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786512017613366-32144-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786512017613366-32144-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786512017613366-32144-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.100.62:44577
--webserver_interface=127.31.100.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.62.keytab
--principal=kudu/127.31.100.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.100.20:45353
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.100.62:44577
--ranger_config_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786512017613366-32144-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.31.100.20:41715/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786512017613366-32144-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786512017613366-32144-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786512017613366-32144-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 05:21:59.575177 18369 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 05:21:59.575376 18369 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 05:21:59.575397 18369 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 05:21:59.576534 18369 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 05:21:59.576573 18369 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 05:21:59.576586 18369 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 05:21:59.576597 18369 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 05:21:59.576617 18369 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 05:21:59.578405 18369 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.100.20:45353
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786512017613366-32144-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786512017613366-32144-0/minicluster-data/master-0/wal
--ranger_kms_url=127.31.100.20:41715/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.31.100.62:44577
--ranger_config_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786512017613366-32144-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.62.keytab
--principal=kudu/127.31.100.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.100.62:44577
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786512017613366-32144-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.100.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18369
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786512017613366-32144-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 04:20:34 UTC on ccycloud.quasar-phqdqp.root.comops.site
build id 4581871
I20260812 05:21:59.578631 18369 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 05:21:59.578835 18369 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 05:21:59.583719 18381 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 05:21:59.583729 18376 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 05:21:59.583755 18375 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 05:21:59.583825 18369 server_base.cc:1034] running on GCE node
I20260812 05:21:59.584201 18369 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 05:21:59.584467 18369 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 05:21:59.585611 18369 hybrid_clock.cc:648] HybridClock initialized: now 1786512119585592 us; error 35 us; skew 500 ppm
Aug 12 05:21:59 dist-test-slave-42z9 krb5kdc[17576](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512119, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.100.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 05:21:59.592417 18369 init.cc:377] Logged in from keytab as kudu/127.31.100.62@KRBTEST.COM (short username kudu)
I20260812 05:21:59.592841 18369 webserver.cc:460] Webserver started at http://127.31.100.62:34953/ using document root <none> and password file <none>
I20260812 05:21:59.593015 18369 fs_manager.cc:359] Metadata directory not provided
I20260812 05:21:59.593048 18369 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 05:21:59.593142 18369 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 12 05:21:59 dist-test-slave-42z9 krb5kdc[17576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512119, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.100.62@KRBTEST.COM for HTTP/127.31.100.20@KRBTEST.COM
I20260812 05:21:59.647719 18369 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786512017613366-32144-0/minicluster-data/master-0/data/instance:
uuid: "ae69a8d0e32545859c9bb86697cf7591"
format_stamp: "Formatted at 2026-08-12 05:21:59 on dist-test-slave-42z9"
server_key: "bb1080537f1e433dcb889e07bc7f1f34"
server_key_iv: "aa685f822e504ca081b1e423de1910ca"
server_key_version: "kuduclusterkey@0"
I20260812 05:21:59.648182 18369 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786512017613366-32144-0/minicluster-data/master-0/wal/instance:
uuid: "ae69a8d0e32545859c9bb86697cf7591"
format_stamp: "Formatted at 2026-08-12 05:21:59 on dist-test-slave-42z9"
server_key: "bb1080537f1e433dcb889e07bc7f1f34"
server_key_iv: "aa685f822e504ca081b1e423de1910ca"
server_key_version: "kuduclusterkey@0"
I20260812 05:21:59.650054 18369 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20260812 05:21:59.664391 18390 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 05:21:59.664618 18369 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260812 05:21:59.664729 18369 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786512017613366-32144-0/minicluster-data/master-0/data,/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786512017613366-32144-0/minicluster-data/master-0/wal
uuid: "ae69a8d0e32545859c9bb86697cf7591"
format_stamp: "Formatted at 2026-08-12 05:21:59 on dist-test-slave-42z9"
server_key: "bb1080537f1e433dcb889e07bc7f1f34"
server_key_iv: "aa685f822e504ca081b1e423de1910ca"
server_key_version: "kuduclusterkey@0"
I20260812 05:21:59.664821 18369 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786512017613366-32144-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786512017613366-32144-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786512017613366-32144-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 05:21:59.705603 18369 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 05:21:59.707118 18369 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 05:21:59.707232 18369 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 05:21:59.714870 18369 rpc_server.cc:307] RPC server started. Bound to: 127.31.100.62:44577
I20260812 05:21:59.714969 18483 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.100.62:44577 every 8 connection(s)
I20260812 05:21:59.715317 18369 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786512017613366-32144-0/minicluster-data/master-0/data/info.pb
I20260812 05:21:59.716032 18484 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 05:21:59.716434 32144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0waFvT/build/release/bin/kudu as pid 18369
I20260812 05:21:59.716555 32144 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786512017613366-32144-0/minicluster-data/master-0/wal/instance
I20260812 05:21:59.716754 32144 external_mini_cluster.cc:1428] Setting key 913aaa7955346917e1a2b42d9655351e
I20260812 05:21:59.719959 18484 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ae69a8d0e32545859c9bb86697cf7591: Bootstrap starting.
I20260812 05:21:59.720743 18484 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ae69a8d0e32545859c9bb86697cf7591: Neither blocks nor log segments found. Creating new log.
I20260812 05:21:59.721190 18484 log.cc:824] T 00000000000000000000000000000000 P ae69a8d0e32545859c9bb86697cf7591: Log is configured to *not* fsync() on all Append() calls
I20260812 05:21:59.721956 18484 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ae69a8d0e32545859c9bb86697cf7591: No bootstrap required, opened a new log
Aug 12 05:21:59 dist-test-slave-42z9 krb5kdc[17576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512119, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.100.62@KRBTEST.COM
I20260812 05:21:59.724992 18484 raft_consensus.cc:348] T 00000000000000000000000000000000 P ae69a8d0e32545859c9bb86697cf7591 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae69a8d0e32545859c9bb86697cf7591" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 44577 } }
I20260812 05:21:59.725129 18484 raft_consensus.cc:374] T 00000000000000000000000000000000 P ae69a8d0e32545859c9bb86697cf7591 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 05:21:59.725152 18484 raft_consensus.cc:729] T 00000000000000000000000000000000 P ae69a8d0e32545859c9bb86697cf7591 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae69a8d0e32545859c9bb86697cf7591, State: Initialized, Role: FOLLOWER
I20260812 05:21:59.725258 18484 consensus_queue.cc:260] T 00000000000000000000000000000000 P ae69a8d0e32545859c9bb86697cf7591 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae69a8d0e32545859c9bb86697cf7591" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 44577 } }
I20260812 05:21:59.725328 18484 raft_consensus.cc:388] T 00000000000000000000000000000000 P ae69a8d0e32545859c9bb86697cf7591 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 05:21:59.725360 18484 raft_consensus.cc:482] T 00000000000000000000000000000000 P ae69a8d0e32545859c9bb86697cf7591 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 05:21:59.725406 18484 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ae69a8d0e32545859c9bb86697cf7591 [term 0 FOLLOWER]: Advancing to term 1
I20260812 05:21:59.726102 18484 raft_consensus.cc:504] T 00000000000000000000000000000000 P ae69a8d0e32545859c9bb86697cf7591 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae69a8d0e32545859c9bb86697cf7591" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 44577 } }
I20260812 05:21:59.726207 18484 leader_election.cc:302] T 00000000000000000000000000000000 P ae69a8d0e32545859c9bb86697cf7591 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ae69a8d0e32545859c9bb86697cf7591; no voters:
I20260812 05:21:59.726365 18484 leader_election.cc:288] T 00000000000000000000000000000000 P ae69a8d0e32545859c9bb86697cf7591 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 05:21:59.726475 18490 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ae69a8d0e32545859c9bb86697cf7591 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 05:21:59.726608 18484 sys_catalog.cc:564] T 00000000000000000000000000000000 P ae69a8d0e32545859c9bb86697cf7591 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 05:21:59.726774 18490 raft_consensus.cc:686] T 00000000000000000000000000000000 P ae69a8d0e32545859c9bb86697cf7591 [term 1 LEADER]: Becoming Leader. State: Replica: ae69a8d0e32545859c9bb86697cf7591, State: Running, Role: LEADER
I20260812 05:21:59.726953 18490 consensus_queue.cc:237] T 00000000000000000000000000000000 P ae69a8d0e32545859c9bb86697cf7591 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae69a8d0e32545859c9bb86697cf7591" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 44577 } }
I20260812 05:21:59.727385 18495 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae69a8d0e32545859c9bb86697cf7591 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ae69a8d0e32545859c9bb86697cf7591. Latest consensus state: current_term: 1 leader_uuid: "ae69a8d0e32545859c9bb86697cf7591" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae69a8d0e32545859c9bb86697cf7591" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 44577 } } }
I20260812 05:21:59.727474 18495 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae69a8d0e32545859c9bb86697cf7591 [sys.catalog]: This master's current role is: LEADER
I20260812 05:21:59.727375 18494 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae69a8d0e32545859c9bb86697cf7591 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ae69a8d0e32545859c9bb86697cf7591" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae69a8d0e32545859c9bb86697cf7591" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 44577 } } }
I20260812 05:21:59.727762 18494 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae69a8d0e32545859c9bb86697cf7591 [sys.catalog]: This master's current role is: LEADER
I20260812 05:21:59.728538 18487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:21:59.717304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39496 (local address 127.31.100.62:44577)
0812 05:21:59.717581 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0812 05:21:59.717589 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 05:21:59.717628 (+ 39us) server_negotiation.cc:407] Connection header received
0812 05:21:59.718163 (+ 535us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:21:59.718170 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:21:59.718219 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:21:59.718330 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:21:59.719206 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:21:59.720375 (+ 1169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:21:59.721358 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:21:59.721866 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:21:59.721958 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:21:59.724514 (+ 2556us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 05:21:59.724518 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 05:21:59.724518 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 05:21:59.724528 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0812 05:21:59.726511 (+ 1983us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:21:59.726984 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:21:59.726987 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:21:59.726987 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:21:59.727085 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:21:59.727310 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:21:59.727311 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:21:59.727312 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:21:59.727649 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 05:21:59.727796 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0812 05:21:59.728157 (+ 361us) server_negotiation.cc:299] Negotiation successful
0812 05:21:59.728280 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":221,"thread_start_us":143,"threads_started":1}
I20260812 05:21:59.729116 18484 ranger_client.cc:318] Using new properties file: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786512017613366-32144-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260812 05:21:59.730124 18505 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 05:21:59.734639 18505 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 05:21:59.736900 18505 catalog_manager.cc:1347] Generated new cluster ID: d8376214559e4b93a217d67bcab5be52
I20260812 05:21:59.736953 18505 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 05:21:59.753693 18505 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 05:21:59.754248 18505 catalog_manager.cc:1504] Loading token signing keys...
I20260812 05:21:59.760473 18505 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ae69a8d0e32545859c9bb86697cf7591: Generated new TSK 0
I20260812 05:21:59.760639 18505 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.31.100.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.100.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.100.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.1.keytab.
Entry for principal kudu/127.31.100.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.1.keytab.
I20260812 05:22:00.575840 32144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0waFvT/build/release/bin/kudu
/tmp/dist-test-task0waFvT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786512017613366-32144-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786512017613366-32144-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786512017613366-32144-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786512017613366-32144-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.31.100.20:41715/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.100.1:0
--local_ip_for_outbound_sockets=127.31.100.1
--webserver_interface=127.31.100.1
--webserver_port=0
--tserver_master_addrs=127.31.100.62:44577
--keytab_file=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.1.keytab
--principal=kudu/127.31.100.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.100.20:45353
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786512017613366-32144-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786512017613366-32144-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786512017613366-32144-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 05:22:00.706943 18553 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 05:22:00.707145 18553 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 05:22:00.707176 18553 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 05:22:00.708316 18553 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 05:22:00.708359 18553 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 05:22:00.708391 18553 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.100.1
I20260812 05:22:00.710085 18553 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.100.20:45353
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786512017613366-32144-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786512017613366-32144-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.31.100.20:41715/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.1.keytab
--principal=kudu/127.31.100.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.100.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786512017613366-32144-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.100.1
--webserver_port=0
--tserver_master_addrs=127.31.100.62:44577
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18553
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.100.1
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786512017613366-32144-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 04:20:34 UTC on ccycloud.quasar-phqdqp.root.comops.site
build id 4581871
I20260812 05:22:00.710309 18553 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 05:22:00.710512 18553 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 05:22:00.716627 18565 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 05:22:00.717242 18570 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 05:22:00.717765 18567 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 05:22:00.718374 18553 server_base.cc:1034] running on GCE node
I20260812 05:22:00.718590 18553 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 05:22:00.718815 18553 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 05:22:00.719986 18553 hybrid_clock.cc:648] HybridClock initialized: now 1786512120719976 us; error 36 us; skew 500 ppm
Aug 12 05:22:00 dist-test-slave-42z9 krb5kdc[17576](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512120, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.100.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 05:22:00.722894 18553 init.cc:377] Logged in from keytab as kudu/127.31.100.1@KRBTEST.COM (short username kudu)
I20260812 05:22:00.723328 18553 webserver.cc:460] Webserver started at http://127.31.100.1:42401/ using document root <none> and password file <none>
I20260812 05:22:00.723531 18553 fs_manager.cc:359] Metadata directory not provided
I20260812 05:22:00.723573 18553 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 05:22:00.723693 18553 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 12 05:22:00 dist-test-slave-42z9 krb5kdc[17576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512120, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.100.1@KRBTEST.COM for HTTP/127.31.100.20@KRBTEST.COM
I20260812 05:22:00.738231 18553 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786512017613366-32144-0/minicluster-data/ts-0/data/instance:
uuid: "ffc155eed9214c8a80115ae4d04f4cbe"
format_stamp: "Formatted at 2026-08-12 05:22:00 on dist-test-slave-42z9"
server_key: "e7e8e1f4f26ae2c394310fb61c26a691"
server_key_iv: "ac39993b0238df1d50de6b24c3030337"
server_key_version: "kuduclusterkey@0"
I20260812 05:22:00.738772 18553 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786512017613366-32144-0/minicluster-data/ts-0/wal/instance:
uuid: "ffc155eed9214c8a80115ae4d04f4cbe"
format_stamp: "Formatted at 2026-08-12 05:22:00 on dist-test-slave-42z9"
server_key: "e7e8e1f4f26ae2c394310fb61c26a691"
server_key_iv: "ac39993b0238df1d50de6b24c3030337"
server_key_version: "kuduclusterkey@0"
I20260812 05:22:00.740749 18553 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20260812 05:22:00.755371 18590 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 05:22:00.755579 18553 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260812 05:22:00.755678 18553 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786512017613366-32144-0/minicluster-data/ts-0/data,/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786512017613366-32144-0/minicluster-data/ts-0/wal
uuid: "ffc155eed9214c8a80115ae4d04f4cbe"
format_stamp: "Formatted at 2026-08-12 05:22:00 on dist-test-slave-42z9"
server_key: "e7e8e1f4f26ae2c394310fb61c26a691"
server_key_iv: "ac39993b0238df1d50de6b24c3030337"
server_key_version: "kuduclusterkey@0"
I20260812 05:22:00.755769 18553 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786512017613366-32144-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786512017613366-32144-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786512017613366-32144-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 05:22:00.774459 18553 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 05:22:00.776818 18553 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 05:22:00.777002 18553 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 05:22:00.777364 18553 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 05:22:00.777859 18553 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 05:22:00.777940 18553 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 05:22:00.777989 18553 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 05:22:00.778035 18553 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 05:22:00.791222 18553 rpc_server.cc:307] RPC server started. Bound to: 127.31.100.1:34077
I20260812 05:22:00.791333 18798 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.100.1:34077 every 8 connection(s)
I20260812 05:22:00.791765 18553 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786512017613366-32144-0/minicluster-data/ts-0/data/info.pb
I20260812 05:22:00.798395 32144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0waFvT/build/release/bin/kudu as pid 18553
I20260812 05:22:00.798506 32144 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786512017613366-32144-0/minicluster-data/ts-0/wal/instance
I20260812 05:22:00.798703 32144 external_mini_cluster.cc:1428] Setting key cdc2cbded840c8e9be1b259c360c8cbb
Aug 12 05:22:00 dist-test-slave-42z9 krb5kdc[17576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512120, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.100.1@KRBTEST.COM for kudu/127.31.100.62@KRBTEST.COM
I20260812 05:22:00.803216 18803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:22:00.792850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.100.1:41027 (local address 127.31.100.62:44577)
0812 05:22:00.792965 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0812 05:22:00.792967 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 05:22:00.792978 (+ 11us) server_negotiation.cc:407] Connection header received
0812 05:22:00.793754 (+ 776us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:22:00.793755 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:22:00.793770 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:22:00.793844 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:22:00.794553 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:22:00.795374 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:22:00.796892 (+ 1518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:22:00.797212 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:22:00.797291 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:22:00.799947 (+ 2656us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 05:22:00.799949 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 05:22:00.799949 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 05:22:00.799957 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0812 05:22:00.801689 (+ 1732us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:22:00.802164 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:22:00.802165 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:22:00.802166 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:22:00.802232 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:22:00.802525 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:22:00.802526 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:22:00.802527 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:22:00.802662 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 05:22:00.802724 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0812 05:22:00.803032 (+ 308us) server_negotiation.cc:299] Negotiation successful
0812 05:22:00.803119 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97,"thread_start_us":53,"threads_started":1}
I20260812 05:22:00.803532 18801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:22:00.792548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.62:44577 (local address 127.31.100.1:41027)
0812 05:22:00.792750 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0812 05:22:00.792799 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0812 05:22:00.793647 (+ 848us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:22:00.793926 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:22:00.793931 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:22:00.794172 (+ 241us) client_negotiation.cc:189] Negotiated authn=SASL
0812 05:22:00.794446 (+ 274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:22:00.794453 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:22:00.795471 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:22:00.795472 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:22:00.796793 (+ 1321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:22:00.796796 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:22:00.797300 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:22:00.797301 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:22:00.797521 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:22:00.798196 (+ 675us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 05:22:00.798207 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0812 05:22:00.799846 (+ 1639us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 05:22:00.801787 (+ 1941us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 05:22:00.801794 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 05:22:00.801796 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 05:22:00.802086 (+ 290us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 05:22:00.802292 (+ 206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 05:22:00.802293 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 05:22:00.802293 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 05:22:00.802452 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 05:22:00.802751 (+ 299us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 05:22:00.802757 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 05:22:00.802902 (+ 145us) client_negotiation.cc:769] Sending connection context
0812 05:22:00.802995 (+ 93us) client_negotiation.cc:240] Negotiation successful
0812 05:22:00.803134 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":94,"threads_started":1}
I20260812 05:22:00.804064 18799 heartbeater.cc:347] Connected to a master server at 127.31.100.62:44577
I20260812 05:22:00.804152 18799 heartbeater.cc:464] Registering TS with master...
I20260812 05:22:00.804318 18799 heartbeater.cc:511] Master 127.31.100.62:44577 requested a full tablet report, sending...
I20260812 05:22:00.804841 18420 ts_manager.cc:194] Registered new tserver with Master: ffc155eed9214c8a80115ae4d04f4cbe (127.31.100.1:34077)
I20260812 05:22:00.805783 18420 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.100.1@KRBTEST.COM'} at 127.31.100.1:41027
WARNING: no policy specified for kudu/127.31.100.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.100.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.100.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.2.keytab.
Entry for principal kudu/127.31.100.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.2.keytab.
I20260812 05:22:00.828469 32144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0waFvT/build/release/bin/kudu
/tmp/dist-test-task0waFvT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786512017613366-32144-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786512017613366-32144-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786512017613366-32144-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786512017613366-32144-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.31.100.20:41715/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.100.2:0
--local_ip_for_outbound_sockets=127.31.100.2
--webserver_interface=127.31.100.2
--webserver_port=0
--tserver_master_addrs=127.31.100.62:44577
--keytab_file=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.2.keytab
--principal=kudu/127.31.100.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.100.20:45353
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786512017613366-32144-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786512017613366-32144-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786512017613366-32144-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 05:22:00.957348 18806 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 05:22:00.957726 18806 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 05:22:00.957809 18806 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 05:22:00.959496 18806 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 05:22:00.959602 18806 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 05:22:00.959686 18806 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.100.2
I20260812 05:22:00.961865 18806 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.100.20:45353
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786512017613366-32144-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786512017613366-32144-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.31.100.20:41715/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.2.keytab
--principal=kudu/127.31.100.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.100.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786512017613366-32144-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.100.2
--webserver_port=0
--tserver_master_addrs=127.31.100.62:44577
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18806
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.100.2
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786512017613366-32144-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 04:20:34 UTC on ccycloud.quasar-phqdqp.root.comops.site
build id 4581871
I20260812 05:22:00.962337 18806 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 05:22:00.962661 18806 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 05:22:00.969167 18817 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 05:22:00.969543 18816 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 05:22:00.969741 18822 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 05:22:00.970656 18806 server_base.cc:1034] running on GCE node
I20260812 05:22:00.970971 18806 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 05:22:00.971240 18806 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 05:22:00.972702 18806 hybrid_clock.cc:648] HybridClock initialized: now 1786512120972669 us; error 43 us; skew 500 ppm
Aug 12 05:22:00 dist-test-slave-42z9 krb5kdc[17576](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512120, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.100.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 05:22:00.976157 18806 init.cc:377] Logged in from keytab as kudu/127.31.100.2@KRBTEST.COM (short username kudu)
I20260812 05:22:00.976672 18806 webserver.cc:460] Webserver started at http://127.31.100.2:46789/ using document root <none> and password file <none>
I20260812 05:22:00.976926 18806 fs_manager.cc:359] Metadata directory not provided
I20260812 05:22:00.976985 18806 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 05:22:00.977108 18806 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 12 05:22:00 dist-test-slave-42z9 krb5kdc[17576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512120, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.100.2@KRBTEST.COM for HTTP/127.31.100.20@KRBTEST.COM
I20260812 05:22:01.002231 18806 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786512017613366-32144-0/minicluster-data/ts-1/data/instance:
uuid: "7b308c61c8be4008a4442f3467143f70"
format_stamp: "Formatted at 2026-08-12 05:22:00 on dist-test-slave-42z9"
server_key: "ddd62ada739ddb048e917ef3cf8910a9"
server_key_iv: "ec4aacb5cebb403ba5a6d8b5555d1cd5"
server_key_version: "kuduclusterkey@0"
I20260812 05:22:01.002885 18806 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786512017613366-32144-0/minicluster-data/ts-1/wal/instance:
uuid: "7b308c61c8be4008a4442f3467143f70"
format_stamp: "Formatted at 2026-08-12 05:22:00 on dist-test-slave-42z9"
server_key: "ddd62ada739ddb048e917ef3cf8910a9"
server_key_iv: "ec4aacb5cebb403ba5a6d8b5555d1cd5"
server_key_version: "kuduclusterkey@0"
I20260812 05:22:01.007035 18806 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.000s
I20260812 05:22:01.033596 18833 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 05:22:01.033851 18806 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.001s sys 0.000s
I20260812 05:22:01.034022 18806 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786512017613366-32144-0/minicluster-data/ts-1/data,/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786512017613366-32144-0/minicluster-data/ts-1/wal
uuid: "7b308c61c8be4008a4442f3467143f70"
format_stamp: "Formatted at 2026-08-12 05:22:00 on dist-test-slave-42z9"
server_key: "ddd62ada739ddb048e917ef3cf8910a9"
server_key_iv: "ec4aacb5cebb403ba5a6d8b5555d1cd5"
server_key_version: "kuduclusterkey@0"
I20260812 05:22:01.034211 18806 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786512017613366-32144-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786512017613366-32144-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786512017613366-32144-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
Aug 12 05:22:01 dist-test-slave-42z9 krb5kdc[17576](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786512121, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.100.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 05:22:01.056488 18806 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 05:22:01.059000 18806 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 05:22:01.059230 18806 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 05:22:01.059612 18806 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 05:22:01.060087 18806 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 05:22:01.060182 18806 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 05:22:01.060263 18806 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 05:22:01.060321 18806 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 05:22:01.068051 18806 rpc_server.cc:307] RPC server started. Bound to: 127.31.100.2:44677
I20260812 05:22:01.068576 18806 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786512017613366-32144-0/minicluster-data/ts-1/data/info.pb
I20260812 05:22:01.068727 32144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0waFvT/build/release/bin/kudu as pid 18806
I20260812 05:22:01.068845 32144 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786512017613366-32144-0/minicluster-data/ts-1/wal/instance
I20260812 05:22:01.069077 32144 external_mini_cluster.cc:1428] Setting key f7fc00f059b7f12ea4bb54d9e5a33a83
I20260812 05:22:01.071880 19040 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.100.2:44677 every 8 connection(s)
Aug 12 05:22:01 dist-test-slave-42z9 krb5kdc[17576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512120, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.100.2@KRBTEST.COM for kudu/127.31.100.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.31.100.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.100.3@KRBTEST.COM" created.
I20260812 05:22:01.095626 18803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:22:01.073236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.100.2:46141 (local address 127.31.100.62:44577)
0812 05:22:01.073280 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0812 05:22:01.073282 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 05:22:01.073618 (+ 336us) server_negotiation.cc:407] Connection header received
0812 05:22:01.074264 (+ 646us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:22:01.074265 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:22:01.074282 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:22:01.074313 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:22:01.076943 (+ 2630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:22:01.077758 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:22:01.079297 (+ 1539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:22:01.079625 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:22:01.079672 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:22:01.089616 (+ 9944us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 05:22:01.089618 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 05:22:01.089618 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 05:22:01.089628 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0812 05:22:01.091366 (+ 1738us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:22:01.092047 (+ 681us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:22:01.092049 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:22:01.092049 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:22:01.092108 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:22:01.092537 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:22:01.092539 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:22:01.092539 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:22:01.092707 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 05:22:01.092752 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0812 05:22:01.095424 (+ 2672us) server_negotiation.cc:299] Negotiation successful
0812 05:22:01.095521 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260812 05:22:01.096413 19048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:22:01.072927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.62:44577 (local address 127.31.100.2:46141)
0812 05:22:01.073498 (+ 571us) negotiation.cc:107] Waiting for socket to connect
0812 05:22:01.073541 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0812 05:22:01.074147 (+ 606us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:22:01.074559 (+ 412us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:22:01.074563 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:22:01.074851 (+ 288us) client_negotiation.cc:189] Negotiated authn=SASL
0812 05:22:01.075265 (+ 414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:22:01.075273 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:22:01.077879 (+ 2606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:22:01.077880 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:22:01.079195 (+ 1315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:22:01.079199 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:22:01.079765 (+ 566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:22:01.079766 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:22:01.079945 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:22:01.080596 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 05:22:01.080606 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0812 05:22:01.082392 (+ 1786us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 05:22:01.091630 (+ 9238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 05:22:01.091638 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 05:22:01.091640 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 05:22:01.091955 (+ 315us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 05:22:01.092177 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 05:22:01.092178 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 05:22:01.092178 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 05:22:01.092314 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 05:22:01.092801 (+ 487us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 05:22:01.092808 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 05:22:01.095253 (+ 2445us) client_negotiation.cc:769] Sending connection context
0812 05:22:01.095997 (+ 744us) client_negotiation.cc:240] Negotiation successful
0812 05:22:01.096175 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":532,"thread_start_us":121,"threads_started":1}
I20260812 05:22:01.097085 19041 heartbeater.cc:347] Connected to a master server at 127.31.100.62:44577
I20260812 05:22:01.097179 19041 heartbeater.cc:464] Registering TS with master...
I20260812 05:22:01.097359 19041 heartbeater.cc:511] Master 127.31.100.62:44577 requested a full tablet report, sending...
I20260812 05:22:01.097782 18420 ts_manager.cc:194] Registered new tserver with Master: 7b308c61c8be4008a4442f3467143f70 (127.31.100.2:44677)
I20260812 05:22:01.098275 18420 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.100.2@KRBTEST.COM'} at 127.31.100.2:46141
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.100.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.3.keytab.
Entry for principal kudu/127.31.100.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.3.keytab.
I20260812 05:22:01.116966 32144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0waFvT/build/release/bin/kudu
/tmp/dist-test-task0waFvT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786512017613366-32144-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786512017613366-32144-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786512017613366-32144-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786512017613366-32144-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.31.100.20:41715/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.100.3:0
--local_ip_for_outbound_sockets=127.31.100.3
--webserver_interface=127.31.100.3
--webserver_port=0
--tserver_master_addrs=127.31.100.62:44577
--keytab_file=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.3.keytab
--principal=kudu/127.31.100.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.100.20:45353
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786512017613366-32144-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786512017613366-32144-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786512017613366-32144-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 05:22:01.265535 19059 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 05:22:01.265950 19059 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 05:22:01.266062 19059 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 05:22:01.267897 19059 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 05:22:01.268023 19059 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 05:22:01.268114 19059 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.100.3
I20260812 05:22:01.270654 19059 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.100.20:45353
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786512017613366-32144-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786512017613366-32144-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.31.100.20:41715/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.3.keytab
--principal=kudu/127.31.100.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.100.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786512017613366-32144-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.100.3
--webserver_port=0
--tserver_master_addrs=127.31.100.62:44577
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19059
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.100.3
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786512017613366-32144-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 04:20:34 UTC on ccycloud.quasar-phqdqp.root.comops.site
build id 4581871
I20260812 05:22:01.271100 19059 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 05:22:01.271459 19059 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 05:22:01.278862 19073 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 05:22:01.278905 19071 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 05:22:01.279302 19067 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 05:22:01.280490 19059 server_base.cc:1034] running on GCE node
I20260812 05:22:01.280695 19059 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 05:22:01.280941 19059 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 05:22:01.288298 19059 hybrid_clock.cc:648] HybridClock initialized: now 1786512121288267 us; error 44 us; skew 500 ppm
Aug 12 05:22:01 dist-test-slave-42z9 krb5kdc[17576](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512121, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.100.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 05:22:01.291554 19059 init.cc:377] Logged in from keytab as kudu/127.31.100.3@KRBTEST.COM (short username kudu)
I20260812 05:22:01.292121 19059 webserver.cc:460] Webserver started at http://127.31.100.3:42889/ using document root <none> and password file <none>
I20260812 05:22:01.292402 19059 fs_manager.cc:359] Metadata directory not provided
I20260812 05:22:01.292454 19059 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 05:22:01.292577 19059 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 12 05:22:01 dist-test-slave-42z9 krb5kdc[17576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512121, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.100.3@KRBTEST.COM for HTTP/127.31.100.20@KRBTEST.COM
I20260812 05:22:01.309931 19059 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786512017613366-32144-0/minicluster-data/ts-2/data/instance:
uuid: "13089233e862406cb5dc1f847c76892a"
format_stamp: "Formatted at 2026-08-12 05:22:01 on dist-test-slave-42z9"
server_key: "d4aeee1bf916e74232c8905f0d5c73e1"
server_key_iv: "7b2119840f74d2770cb5d1c1111b5f08"
server_key_version: "kuduclusterkey@0"
I20260812 05:22:01.310563 19059 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786512017613366-32144-0/minicluster-data/ts-2/wal/instance:
uuid: "13089233e862406cb5dc1f847c76892a"
format_stamp: "Formatted at 2026-08-12 05:22:01 on dist-test-slave-42z9"
server_key: "d4aeee1bf916e74232c8905f0d5c73e1"
server_key_iv: "7b2119840f74d2770cb5d1c1111b5f08"
server_key_version: "kuduclusterkey@0"
I20260812 05:22:01.317638 19059 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.001s sys 0.001s
I20260812 05:22:01.335152 19083 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 05:22:01.335405 19059 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260812 05:22:01.335508 19059 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786512017613366-32144-0/minicluster-data/ts-2/data,/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786512017613366-32144-0/minicluster-data/ts-2/wal
uuid: "13089233e862406cb5dc1f847c76892a"
format_stamp: "Formatted at 2026-08-12 05:22:01 on dist-test-slave-42z9"
server_key: "d4aeee1bf916e74232c8905f0d5c73e1"
server_key_iv: "7b2119840f74d2770cb5d1c1111b5f08"
server_key_version: "kuduclusterkey@0"
I20260812 05:22:01.335602 19059 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786512017613366-32144-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786512017613366-32144-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786512017613366-32144-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 05:22:01.368462 19059 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 05:22:01.372063 19059 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 05:22:01.372273 19059 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 05:22:01.372529 19059 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 05:22:01.372954 19059 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 05:22:01.373062 19059 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 05:22:01.373139 19059 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 05:22:01.373199 19059 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 05:22:01.383895 19059 rpc_server.cc:307] RPC server started. Bound to: 127.31.100.3:37899
I20260812 05:22:01.384415 19059 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786512017613366-32144-0/minicluster-data/ts-2/data/info.pb
I20260812 05:22:01.384943 19287 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.100.3:37899 every 8 connection(s)
I20260812 05:22:01.392344 32144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0waFvT/build/release/bin/kudu as pid 19059
I20260812 05:22:01.392459 32144 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786512017613366-32144-0/minicluster-data/ts-2/wal/instance
I20260812 05:22:01.392644 32144 external_mini_cluster.cc:1428] Setting key fe84c431d33ccd6818e2ba75277659cb
Aug 12 05:22:01 dist-test-slave-42z9 krb5kdc[17576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512121, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.100.3@KRBTEST.COM for kudu/127.31.100.62@KRBTEST.COM
I20260812 05:22:01.396881 18803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:22:01.385501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.100.3:41123 (local address 127.31.100.62:44577)
0812 05:22:01.388898 (+ 3397us) server_negotiation.cc:206] Beginning negotiation
0812 05:22:01.388900 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 05:22:01.388910 (+ 10us) server_negotiation.cc:407] Connection header received
0812 05:22:01.388937 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:22:01.388938 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:22:01.388948 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:22:01.389008 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:22:01.389632 (+ 624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:22:01.390144 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:22:01.391622 (+ 1478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:22:01.391813 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:22:01.391865 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:22:01.394454 (+ 2589us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 05:22:01.394456 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 05:22:01.394456 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 05:22:01.394463 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 05:22:01.395647 (+ 1184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:22:01.396007 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:22:01.396008 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:22:01.396008 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:22:01.396058 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:22:01.396277 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:22:01.396278 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:22:01.396279 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:22:01.396383 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 05:22:01.396435 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0812 05:22:01.396714 (+ 279us) server_negotiation.cc:299] Negotiation successful
0812 05:22:01.396794 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3354}
I20260812 05:22:01.397032 19290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:22:01.385351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.62:44577 (local address 127.31.100.3:41123)
0812 05:22:01.385518 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0812 05:22:01.385561 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0812 05:22:01.386032 (+ 471us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:22:01.389107 (+ 3075us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:22:01.389111 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:22:01.389296 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0812 05:22:01.389545 (+ 249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:22:01.389551 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:22:01.390225 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:22:01.390226 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:22:01.391536 (+ 1310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:22:01.391539 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:22:01.391885 (+ 346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:22:01.391886 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:22:01.392075 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:22:01.392734 (+ 659us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 05:22:01.392745 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0812 05:22:01.394368 (+ 1623us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 05:22:01.395715 (+ 1347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 05:22:01.395722 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 05:22:01.395724 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 05:22:01.395933 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 05:22:01.396108 (+ 175us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 05:22:01.396108 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 05:22:01.396109 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 05:22:01.396216 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 05:22:01.396435 (+ 219us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 05:22:01.396441 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 05:22:01.396581 (+ 140us) client_negotiation.cc:769] Sending connection context
0812 05:22:01.396675 (+ 94us) client_negotiation.cc:240] Negotiation successful
0812 05:22:01.396816 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":65,"threads_started":1}
I20260812 05:22:01.397562 19288 heartbeater.cc:347] Connected to a master server at 127.31.100.62:44577
I20260812 05:22:01.397671 19288 heartbeater.cc:464] Registering TS with master...
I20260812 05:22:01.397867 19288 heartbeater.cc:511] Master 127.31.100.62:44577 requested a full tablet report, sending...
I20260812 05:22:01.398216 18420 ts_manager.cc:194] Registered new tserver with Master: 13089233e862406cb5dc1f847c76892a (127.31.100.3:37899)
I20260812 05:22:01.398599 18420 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.100.3@KRBTEST.COM'} at 127.31.100.3:41123
I20260812 05:22:01.404562 32144 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 05:22:01.412315 18803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:22:01.405964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39508 (local address 127.31.100.62:44577)
0812 05:22:01.406020 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0812 05:22:01.406022 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 05:22:01.406101 (+ 79us) server_negotiation.cc:407] Connection header received
0812 05:22:01.406140 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:22:01.406141 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:22:01.406151 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:22:01.406215 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:22:01.406696 (+ 481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:22:01.407421 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:22:01.408104 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:22:01.408385 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:22:01.408471 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:22:01.409627 (+ 1156us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 05:22:01.409629 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 05:22:01.409629 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 05:22:01.409637 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0812 05:22:01.411253 (+ 1616us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:22:01.411608 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:22:01.411609 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:22:01.411609 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:22:01.411673 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:22:01.411834 (+ 161us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:22:01.411834 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:22:01.411835 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:22:01.411963 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 05:22:01.412060 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0812 05:22:01.412145 (+ 85us) server_negotiation.cc:299] Negotiation successful
0812 05:22:01.412223 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260812 05:22:01.414031 18420 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39508:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 05:22:01.416428 18420 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 05:22:01.423712 19317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:22:01.419323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.1:34077 (local address 127.0.0.1:60614)
0812 05:22:01.419792 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0812 05:22:01.419801 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0812 05:22:01.419849 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:22:01.420037 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:22:01.420038 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:22:01.420040 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 05:22:01.420082 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:22:01.420084 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:22:01.420966 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:22:01.420967 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:22:01.422428 (+ 1461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:22:01.422430 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:22:01.423424 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:22:01.423425 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:22:01.423502 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:22:01.423516 (+ 14us) client_negotiation.cc:769] Sending connection context
0812 05:22:01.423587 (+ 71us) client_negotiation.cc:240] Negotiation successful
0812 05:22:01.423641 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":433,"mutex_wait_us":23,"thread_start_us":72,"threads_started":1}
I20260812 05:22:01.424044 19312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:22:01.419080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42408 (local address 127.31.100.2:44677)
0812 05:22:01.419284 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0812 05:22:01.419289 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 05:22:01.419475 (+ 186us) server_negotiation.cc:407] Connection header received
0812 05:22:01.419597 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:22:01.419600 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:22:01.419617 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:22:01.419706 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 05:22:01.419970 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:22:01.421117 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:22:01.422496 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:22:01.423358 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:22:01.423429 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:22:01.423580 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0812 05:22:01.423880 (+ 300us) server_negotiation.cc:299] Negotiation successful
0812 05:22:01.423922 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":173,"thread_start_us":91,"threads_started":1}
I20260812 05:22:01.423815 19308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:22:01.418943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.2:44677 (local address 127.0.0.1:42408)
0812 05:22:01.419351 (+ 408us) negotiation.cc:107] Waiting for socket to connect
0812 05:22:01.419408 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0812 05:22:01.419488 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:22:01.419810 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:22:01.419814 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:22:01.419816 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 05:22:01.419882 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:22:01.419884 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:22:01.421211 (+ 1327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:22:01.421212 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:22:01.422315 (+ 1103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:22:01.422318 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:22:01.423614 (+ 1296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:22:01.423615 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:22:01.423668 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:22:01.423672 (+ 4us) client_negotiation.cc:769] Sending connection context
0812 05:22:01.423714 (+ 42us) client_negotiation.cc:240] Negotiation successful
0812 05:22:01.423751 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":379,"thread_start_us":75,"threads_started":1}
I20260812 05:22:01.424535 19318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:22:01.419450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60614 (local address 127.31.100.1:34077)
0812 05:22:01.419577 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0812 05:22:01.419582 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 05:22:01.419903 (+ 321us) server_negotiation.cc:407] Connection header received
0812 05:22:01.419940 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:22:01.419943 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:22:01.419957 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:22:01.420029 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 05:22:01.420134 (+ 105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:22:01.420888 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:22:01.422513 (+ 1625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:22:01.423247 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:22:01.423322 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:22:01.423387 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0812 05:22:01.424415 (+ 1028us) server_negotiation.cc:299] Negotiation successful
0812 05:22:01.424463 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92,"thread_start_us":61,"threads_started":1}
I20260812 05:22:01.425081 18913 tablet_service.cc:1467] Processing CreateTablet for tablet ec886a8283e7439cbde13abaac72179d (DEFAULT_TABLE table=test-table [id=214d70554e4b4c0fb0ec1123528979d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 05:22:01.425253 18676 tablet_service.cc:1467] Processing CreateTablet for tablet ec886a8283e7439cbde13abaac72179d (DEFAULT_TABLE table=test-table [id=214d70554e4b4c0fb0ec1123528979d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 05:22:01.425523 19313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:22:01.419170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.3:37899 (local address 127.0.0.1:33844)
0812 05:22:01.419598 (+ 428us) negotiation.cc:107] Waiting for socket to connect
0812 05:22:01.419606 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0812 05:22:01.419658 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:22:01.421755 (+ 2097us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:22:01.421756 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:22:01.421758 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 05:22:01.421800 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:22:01.421802 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:22:01.423079 (+ 1277us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:22:01.423080 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:22:01.424174 (+ 1094us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:22:01.424177 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:22:01.425143 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:22:01.425144 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:22:01.425219 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:22:01.425223 (+ 4us) client_negotiation.cc:769] Sending connection context
0812 05:22:01.425326 (+ 103us) client_negotiation.cc:240] Negotiation successful
0812 05:22:01.425368 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":414,"thread_start_us":61,"threads_started":1}
I20260812 05:22:01.425522 18913 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ec886a8283e7439cbde13abaac72179d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 05:22:01.425635 18676 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ec886a8283e7439cbde13abaac72179d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 05:22:01.425851 19314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:22:01.419278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33844 (local address 127.31.100.3:37899)
0812 05:22:01.421567 (+ 2289us) server_negotiation.cc:206] Beginning negotiation
0812 05:22:01.421572 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 05:22:01.421582 (+ 10us) server_negotiation.cc:407] Connection header received
0812 05:22:01.421622 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:22:01.421625 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:22:01.421640 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:22:01.421873 (+ 233us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 05:22:01.421944 (+ 71us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:22:01.422973 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:22:01.424302 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:22:01.425058 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:22:01.425130 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:22:01.425201 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0812 05:22:01.425681 (+ 480us) server_negotiation.cc:299] Negotiation successful
0812 05:22:01.425724 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2251,"thread_start_us":94,"threads_started":1}
I20260812 05:22:01.429045 19319 tablet_bootstrap.cc:492] T ec886a8283e7439cbde13abaac72179d P ffc155eed9214c8a80115ae4d04f4cbe: Bootstrap starting.
I20260812 05:22:01.429122 19320 tablet_bootstrap.cc:492] T ec886a8283e7439cbde13abaac72179d P 7b308c61c8be4008a4442f3467143f70: Bootstrap starting.
I20260812 05:22:01.429764 19174 tablet_service.cc:1467] Processing CreateTablet for tablet ec886a8283e7439cbde13abaac72179d (DEFAULT_TABLE table=test-table [id=214d70554e4b4c0fb0ec1123528979d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 05:22:01.430114 19174 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ec886a8283e7439cbde13abaac72179d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 05:22:01.430318 19320 tablet_bootstrap.cc:654] T ec886a8283e7439cbde13abaac72179d P 7b308c61c8be4008a4442f3467143f70: Neither blocks nor log segments found. Creating new log.
I20260812 05:22:01.430840 19319 tablet_bootstrap.cc:654] T ec886a8283e7439cbde13abaac72179d P ffc155eed9214c8a80115ae4d04f4cbe: Neither blocks nor log segments found. Creating new log.
I20260812 05:22:01.430968 19320 log.cc:824] T ec886a8283e7439cbde13abaac72179d P 7b308c61c8be4008a4442f3467143f70: Log is configured to *not* fsync() on all Append() calls
I20260812 05:22:01.431631 19319 log.cc:824] T ec886a8283e7439cbde13abaac72179d P ffc155eed9214c8a80115ae4d04f4cbe: Log is configured to *not* fsync() on all Append() calls
I20260812 05:22:01.431937 19320 tablet_bootstrap.cc:492] T ec886a8283e7439cbde13abaac72179d P 7b308c61c8be4008a4442f3467143f70: No bootstrap required, opened a new log
I20260812 05:22:01.432015 19320 ts_tablet_manager.cc:1397] T ec886a8283e7439cbde13abaac72179d P 7b308c61c8be4008a4442f3467143f70: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260812 05:22:01.434095 19319 tablet_bootstrap.cc:492] T ec886a8283e7439cbde13abaac72179d P ffc155eed9214c8a80115ae4d04f4cbe: No bootstrap required, opened a new log
I20260812 05:22:01.434165 19319 ts_tablet_manager.cc:1397] T ec886a8283e7439cbde13abaac72179d P ffc155eed9214c8a80115ae4d04f4cbe: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260812 05:22:01.434593 19324 tablet_bootstrap.cc:492] T ec886a8283e7439cbde13abaac72179d P 13089233e862406cb5dc1f847c76892a: Bootstrap starting.
I20260812 05:22:01.435361 19320 raft_consensus.cc:348] T ec886a8283e7439cbde13abaac72179d P 7b308c61c8be4008a4442f3467143f70 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b308c61c8be4008a4442f3467143f70" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 44677 } } peers { permanent_uuid: "13089233e862406cb5dc1f847c76892a" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 37899 } } peers { permanent_uuid: "ffc155eed9214c8a80115ae4d04f4cbe" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 34077 } }
I20260812 05:22:01.435541 19320 raft_consensus.cc:374] T ec886a8283e7439cbde13abaac72179d P 7b308c61c8be4008a4442f3467143f70 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 05:22:01.435577 19320 raft_consensus.cc:729] T ec886a8283e7439cbde13abaac72179d P 7b308c61c8be4008a4442f3467143f70 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b308c61c8be4008a4442f3467143f70, State: Initialized, Role: FOLLOWER
I20260812 05:22:01.435695 19320 consensus_queue.cc:260] T ec886a8283e7439cbde13abaac72179d P 7b308c61c8be4008a4442f3467143f70 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b308c61c8be4008a4442f3467143f70" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 44677 } } peers { permanent_uuid: "13089233e862406cb5dc1f847c76892a" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 37899 } } peers { permanent_uuid: "ffc155eed9214c8a80115ae4d04f4cbe" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 34077 } }
I20260812 05:22:01.435863 19324 tablet_bootstrap.cc:654] T ec886a8283e7439cbde13abaac72179d P 13089233e862406cb5dc1f847c76892a: Neither blocks nor log segments found. Creating new log.
I20260812 05:22:01.435956 19320 ts_tablet_manager.cc:1428] T ec886a8283e7439cbde13abaac72179d P 7b308c61c8be4008a4442f3467143f70: Time spent starting tablet: real 0.004s user 0.000s sys 0.004s
I20260812 05:22:01.436072 19041 heartbeater.cc:503] Master 127.31.100.62:44577 was elected leader, sending a full tablet report...
I20260812 05:22:01.436486 19324 log.cc:824] T ec886a8283e7439cbde13abaac72179d P 13089233e862406cb5dc1f847c76892a: Log is configured to *not* fsync() on all Append() calls
I20260812 05:22:01.437536 19324 tablet_bootstrap.cc:492] T ec886a8283e7439cbde13abaac72179d P 13089233e862406cb5dc1f847c76892a: No bootstrap required, opened a new log
I20260812 05:22:01.437603 19324 ts_tablet_manager.cc:1397] T ec886a8283e7439cbde13abaac72179d P 13089233e862406cb5dc1f847c76892a: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260812 05:22:01.438364 19319 raft_consensus.cc:348] T ec886a8283e7439cbde13abaac72179d P ffc155eed9214c8a80115ae4d04f4cbe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b308c61c8be4008a4442f3467143f70" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 44677 } } peers { permanent_uuid: "13089233e862406cb5dc1f847c76892a" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 37899 } } peers { permanent_uuid: "ffc155eed9214c8a80115ae4d04f4cbe" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 34077 } }
I20260812 05:22:01.438544 19319 raft_consensus.cc:374] T ec886a8283e7439cbde13abaac72179d P ffc155eed9214c8a80115ae4d04f4cbe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 05:22:01.438573 19319 raft_consensus.cc:729] T ec886a8283e7439cbde13abaac72179d P ffc155eed9214c8a80115ae4d04f4cbe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ffc155eed9214c8a80115ae4d04f4cbe, State: Initialized, Role: FOLLOWER
I20260812 05:22:01.438678 19319 consensus_queue.cc:260] T ec886a8283e7439cbde13abaac72179d P ffc155eed9214c8a80115ae4d04f4cbe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b308c61c8be4008a4442f3467143f70" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 44677 } } peers { permanent_uuid: "13089233e862406cb5dc1f847c76892a" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 37899 } } peers { permanent_uuid: "ffc155eed9214c8a80115ae4d04f4cbe" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 34077 } }
I20260812 05:22:01.438993 19319 ts_tablet_manager.cc:1428] T ec886a8283e7439cbde13abaac72179d P ffc155eed9214c8a80115ae4d04f4cbe: Time spent starting tablet: real 0.005s user 0.000s sys 0.005s
I20260812 05:22:01.439107 18799 heartbeater.cc:503] Master 127.31.100.62:44577 was elected leader, sending a full tablet report...
I20260812 05:22:01.443544 19324 raft_consensus.cc:348] T ec886a8283e7439cbde13abaac72179d P 13089233e862406cb5dc1f847c76892a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b308c61c8be4008a4442f3467143f70" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 44677 } } peers { permanent_uuid: "13089233e862406cb5dc1f847c76892a" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 37899 } } peers { permanent_uuid: "ffc155eed9214c8a80115ae4d04f4cbe" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 34077 } }
I20260812 05:22:01.443737 19324 raft_consensus.cc:374] T ec886a8283e7439cbde13abaac72179d P 13089233e862406cb5dc1f847c76892a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 05:22:01.443795 19324 raft_consensus.cc:729] T ec886a8283e7439cbde13abaac72179d P 13089233e862406cb5dc1f847c76892a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13089233e862406cb5dc1f847c76892a, State: Initialized, Role: FOLLOWER
I20260812 05:22:01.443909 19324 consensus_queue.cc:260] T ec886a8283e7439cbde13abaac72179d P 13089233e862406cb5dc1f847c76892a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b308c61c8be4008a4442f3467143f70" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 44677 } } peers { permanent_uuid: "13089233e862406cb5dc1f847c76892a" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 37899 } } peers { permanent_uuid: "ffc155eed9214c8a80115ae4d04f4cbe" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 34077 } }
I20260812 05:22:01.444188 19324 ts_tablet_manager.cc:1428] T ec886a8283e7439cbde13abaac72179d P 13089233e862406cb5dc1f847c76892a: Time spent starting tablet: real 0.007s user 0.000s sys 0.003s
I20260812 05:22:01.444309 19288 heartbeater.cc:503] Master 127.31.100.62:44577 was elected leader, sending a full tablet report...
W20260812 05:22:01.544191 18800 tablet.cc:2369] T ec886a8283e7439cbde13abaac72179d P ffc155eed9214c8a80115ae4d04f4cbe: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 05:22:01.573768 19045 tablet.cc:2369] T ec886a8283e7439cbde13abaac72179d P 7b308c61c8be4008a4442f3467143f70: Can't schedule compaction. Clean time has not been advanced past its initial value.
Aug 12 05:22:01 dist-test-slave-42z9 krb5kdc[17576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512121, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.100.62@KRBTEST.COM for HTTP/127.31.100.20@KRBTEST.COM
I20260812 05:22:01.591727 19333 raft_consensus.cc:482] T ec886a8283e7439cbde13abaac72179d P 13089233e862406cb5dc1f847c76892a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 05:22:01.591869 19333 raft_consensus.cc:504] T ec886a8283e7439cbde13abaac72179d P 13089233e862406cb5dc1f847c76892a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b308c61c8be4008a4442f3467143f70" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 44677 } } peers { permanent_uuid: "13089233e862406cb5dc1f847c76892a" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 37899 } } peers { permanent_uuid: "ffc155eed9214c8a80115ae4d04f4cbe" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 34077 } }
I20260812 05:22:01.592236 19333 leader_election.cc:288] T ec886a8283e7439cbde13abaac72179d P 13089233e862406cb5dc1f847c76892a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7b308c61c8be4008a4442f3467143f70 (127.31.100.2:44677), ffc155eed9214c8a80115ae4d04f4cbe (127.31.100.1:34077)
I20260812 05:22:01.596706 19290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:22:01.592480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.1:34077 (local address 127.31.100.3:48415)
0812 05:22:01.592787 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0812 05:22:01.592797 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0812 05:22:01.592842 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:22:01.593248 (+ 406us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:22:01.593250 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:22:01.593251 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 05:22:01.593398 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:22:01.593399 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:22:01.594482 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:22:01.594483 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:22:01.595588 (+ 1105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:22:01.595591 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:22:01.596403 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:22:01.596404 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:22:01.596530 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:22:01.596534 (+ 4us) client_negotiation.cc:769] Sending connection context
0812 05:22:01.596587 (+ 53us) client_negotiation.cc:240] Negotiation successful
0812 05:22:01.596632 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":288,"thread_start_us":203,"threads_started":1}
I20260812 05:22:01.597174 19318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:22:01.593048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.100.3:48415 (local address 127.31.100.1:34077)
0812 05:22:01.593110 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0812 05:22:01.593112 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 05:22:01.593120 (+ 8us) server_negotiation.cc:407] Connection header received
0812 05:22:01.593146 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:22:01.593147 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:22:01.593156 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:22:01.593201 (+ 45us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 05:22:01.593659 (+ 458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:22:01.594387 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:22:01.595765 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:22:01.596327 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:22:01.596404 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:22:01.596413 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0812 05:22:01.597059 (+ 646us) server_negotiation.cc:299] Negotiation successful
0812 05:22:01.597111 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260812 05:22:01.597594 18709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec886a8283e7439cbde13abaac72179d" candidate_uuid: "13089233e862406cb5dc1f847c76892a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ffc155eed9214c8a80115ae4d04f4cbe" is_pre_election: true
I20260812 05:22:01.597759 18709 raft_consensus.cc:2454] T ec886a8283e7439cbde13abaac72179d P ffc155eed9214c8a80115ae4d04f4cbe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 13089233e862406cb5dc1f847c76892a in term 0.
I20260812 05:22:01.598026 19088 leader_election.cc:302] T ec886a8283e7439cbde13abaac72179d P 13089233e862406cb5dc1f847c76892a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 13089233e862406cb5dc1f847c76892a, ffc155eed9214c8a80115ae4d04f4cbe; no voters:
I20260812 05:22:01.598186 19333 raft_consensus.cc:2781] T ec886a8283e7439cbde13abaac72179d P 13089233e862406cb5dc1f847c76892a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 05:22:01.598244 19333 raft_consensus.cc:482] T ec886a8283e7439cbde13abaac72179d P 13089233e862406cb5dc1f847c76892a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 05:22:01.598275 19333 raft_consensus.cc:3037] T ec886a8283e7439cbde13abaac72179d P 13089233e862406cb5dc1f847c76892a [term 0 FOLLOWER]: Advancing to term 1
I20260812 05:22:01.599371 19333 raft_consensus.cc:504] T ec886a8283e7439cbde13abaac72179d P 13089233e862406cb5dc1f847c76892a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b308c61c8be4008a4442f3467143f70" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 44677 } } peers { permanent_uuid: "13089233e862406cb5dc1f847c76892a" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 37899 } } peers { permanent_uuid: "ffc155eed9214c8a80115ae4d04f4cbe" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 34077 } }
I20260812 05:22:01.599506 19333 leader_election.cc:288] T ec886a8283e7439cbde13abaac72179d P 13089233e862406cb5dc1f847c76892a [CANDIDATE]: Term 1 election: Requested vote from peers 7b308c61c8be4008a4442f3467143f70 (127.31.100.2:44677), ffc155eed9214c8a80115ae4d04f4cbe (127.31.100.1:34077)
I20260812 05:22:01.599684 18709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec886a8283e7439cbde13abaac72179d" candidate_uuid: "13089233e862406cb5dc1f847c76892a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ffc155eed9214c8a80115ae4d04f4cbe"
I20260812 05:22:01.599758 18709 raft_consensus.cc:3037] T ec886a8283e7439cbde13abaac72179d P ffc155eed9214c8a80115ae4d04f4cbe [term 0 FOLLOWER]: Advancing to term 1
I20260812 05:22:01.600998 18709 raft_consensus.cc:2454] T ec886a8283e7439cbde13abaac72179d P ffc155eed9214c8a80115ae4d04f4cbe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 13089233e862406cb5dc1f847c76892a in term 1.
I20260812 05:22:01.601184 19088 leader_election.cc:302] T ec886a8283e7439cbde13abaac72179d P 13089233e862406cb5dc1f847c76892a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 13089233e862406cb5dc1f847c76892a, ffc155eed9214c8a80115ae4d04f4cbe; no voters:
I20260812 05:22:01.601261 19333 raft_consensus.cc:2781] T ec886a8283e7439cbde13abaac72179d P 13089233e862406cb5dc1f847c76892a [term 1 FOLLOWER]: Leader election won for term 1
I20260812 05:22:01.601405 19333 raft_consensus.cc:686] T ec886a8283e7439cbde13abaac72179d P 13089233e862406cb5dc1f847c76892a [term 1 LEADER]: Becoming Leader. State: Replica: 13089233e862406cb5dc1f847c76892a, State: Running, Role: LEADER
I20260812 05:22:01.601549 19333 consensus_queue.cc:237] T ec886a8283e7439cbde13abaac72179d P 13089233e862406cb5dc1f847c76892a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b308c61c8be4008a4442f3467143f70" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 44677 } } peers { permanent_uuid: "13089233e862406cb5dc1f847c76892a" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 37899 } } peers { permanent_uuid: "ffc155eed9214c8a80115ae4d04f4cbe" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 34077 } }
I20260812 05:22:01.602339 18420 catalog_manager.cc:5495] T ec886a8283e7439cbde13abaac72179d P 13089233e862406cb5dc1f847c76892a reported cstate change: term changed from 0 to 1, leader changed from <none> to 13089233e862406cb5dc1f847c76892a (127.31.100.3). New cstate: current_term: 1 leader_uuid: "13089233e862406cb5dc1f847c76892a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b308c61c8be4008a4442f3467143f70" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 44677 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "13089233e862406cb5dc1f847c76892a" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 37899 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ffc155eed9214c8a80115ae4d04f4cbe" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 34077 } health_report { overall_health: UNKNOWN } } }
I20260812 05:22:01.602751 19312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:22:01.592976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.100.3:36239 (local address 127.31.100.2:44677)
0812 05:22:01.593098 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0812 05:22:01.593100 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 05:22:01.593108 (+ 8us) server_negotiation.cc:407] Connection header received
0812 05:22:01.593134 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:22:01.593135 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:22:01.593144 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:22:01.593193 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 05:22:01.594336 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:22:01.594956 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:22:01.601804 (+ 6848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:22:01.602309 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:22:01.602355 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:22:01.602364 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0812 05:22:01.602636 (+ 272us) server_negotiation.cc:299] Negotiation successful
0812 05:22:01.602684 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100}
I20260812 05:22:01.603055 19335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:22:01.592385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.2:44677 (local address 127.31.100.3:36239)
0812 05:22:01.592808 (+ 423us) negotiation.cc:107] Waiting for socket to connect
0812 05:22:01.592820 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0812 05:22:01.592875 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:22:01.593521 (+ 646us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:22:01.593523 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:22:01.593525 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 05:22:01.593657 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:22:01.593659 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:22:01.595052 (+ 1393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:22:01.595052 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:22:01.596085 (+ 1033us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:22:01.596087 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:22:01.602423 (+ 6336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:22:01.602424 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:22:01.602544 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:22:01.602548 (+ 4us) client_negotiation.cc:769] Sending connection context
0812 05:22:01.602853 (+ 305us) client_negotiation.cc:240] Negotiation successful
0812 05:22:01.602905 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":341}
I20260812 05:22:01.603583 18942 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec886a8283e7439cbde13abaac72179d" candidate_uuid: "13089233e862406cb5dc1f847c76892a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b308c61c8be4008a4442f3467143f70" is_pre_election: true
I20260812 05:22:01.603739 18942 raft_consensus.cc:2454] T ec886a8283e7439cbde13abaac72179d P 7b308c61c8be4008a4442f3467143f70 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 13089233e862406cb5dc1f847c76892a in term 0.
I20260812 05:22:01.603844 18941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec886a8283e7439cbde13abaac72179d" candidate_uuid: "13089233e862406cb5dc1f847c76892a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b308c61c8be4008a4442f3467143f70"
I20260812 05:22:01.603915 18941 raft_consensus.cc:3037] T ec886a8283e7439cbde13abaac72179d P 7b308c61c8be4008a4442f3467143f70 [term 0 FOLLOWER]: Advancing to term 1
I20260812 05:22:01.604687 18941 raft_consensus.cc:2454] T ec886a8283e7439cbde13abaac72179d P 7b308c61c8be4008a4442f3467143f70 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 13089233e862406cb5dc1f847c76892a in term 1.
W20260812 05:22:01.635349 19289 tablet.cc:2369] T ec886a8283e7439cbde13abaac72179d P 13089233e862406cb5dc1f847c76892a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 05:22:01.651604 19314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:22:01.648192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33848 (local address 127.31.100.3:37899)
0812 05:22:01.648248 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0812 05:22:01.648250 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 05:22:01.648259 (+ 9us) server_negotiation.cc:407] Connection header received
0812 05:22:01.648299 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:22:01.648300 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:22:01.648321 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:22:01.648405 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 05:22:01.648575 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:22:01.649353 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:22:01.650498 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:22:01.650811 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:22:01.650871 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:22:01.651079 (+ 208us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 05:22:01.651210 (+ 131us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 05:22:01.651281 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0812 05:22:01.651501 (+ 220us) server_negotiation.cc:299] Negotiation successful
0812 05:22:01.651522 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260812 05:22:01.654031 18941 raft_consensus.cc:1261] T ec886a8283e7439cbde13abaac72179d P 7b308c61c8be4008a4442f3467143f70 [term 1 FOLLOWER]: Refusing update from remote peer 13089233e862406cb5dc1f847c76892a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 05:22:01.654074 18709 raft_consensus.cc:1261] T ec886a8283e7439cbde13abaac72179d P ffc155eed9214c8a80115ae4d04f4cbe [term 1 FOLLOWER]: Refusing update from remote peer 13089233e862406cb5dc1f847c76892a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 05:22:01.654417 19336 consensus_queue.cc:1035] T ec886a8283e7439cbde13abaac72179d P 13089233e862406cb5dc1f847c76892a [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b308c61c8be4008a4442f3467143f70" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 44677 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 05:22:01.654556 19333 consensus_queue.cc:1035] T ec886a8283e7439cbde13abaac72179d P 13089233e862406cb5dc1f847c76892a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ffc155eed9214c8a80115ae4d04f4cbe" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 34077 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 05:22:01.664186 19339 mvcc.cc:205] Tried to move back new op lower bound from 7317553650291523584 to 7317553650080391168. Current Snapshot: MvccSnapshot[applied={T|T < 7317553650291523584}]
I20260812 05:22:01.664409 19340 mvcc.cc:205] Tried to move back new op lower bound from 7317553650291523584 to 7317553650080391168. Current Snapshot: MvccSnapshot[applied={T|T < 7317553650291523584}]
I20260812 05:22:01.667063 19348 mvcc.cc:205] Tried to move back new op lower bound from 7317553650291523584 to 7317553650080391168. Current Snapshot: MvccSnapshot[applied={T|T < 7317553650291523584}]
I20260812 05:22:01.669639 18420 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39508:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 05:22:01.669737 18420 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39508:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 05:22:01.670583 18420 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ae69a8d0e32545859c9bb86697cf7591: Sending DeleteTablet for 3 replicas of tablet ec886a8283e7439cbde13abaac72179d
I20260812 05:22:01.671229 18913 tablet_service.cc:1514] Processing DeleteTablet for tablet ec886a8283e7439cbde13abaac72179d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 05:22:01 UTC) from {username='kudu', principal='kudu/127.31.100.62@KRBTEST.COM'} at 127.0.0.1:42408
I20260812 05:22:01.671542 19174 tablet_service.cc:1514] Processing DeleteTablet for tablet ec886a8283e7439cbde13abaac72179d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 05:22:01 UTC) from {username='kudu', principal='kudu/127.31.100.62@KRBTEST.COM'} at 127.0.0.1:33844
I20260812 05:22:01.671814 18676 tablet_service.cc:1514] Processing DeleteTablet for tablet ec886a8283e7439cbde13abaac72179d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 05:22:01 UTC) from {username='kudu', principal='kudu/127.31.100.62@KRBTEST.COM'} at 127.0.0.1:60614
I20260812 05:22:01.672379 32144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0waFvT/build/release/bin/kudu with pid 18553
I20260812 05:22:01.689563 19358 tablet_replica.cc:330] T ec886a8283e7439cbde13abaac72179d P 7b308c61c8be4008a4442f3467143f70: stopping tablet replica
I20260812 05:22:01.689641 19360 tablet_replica.cc:330] T ec886a8283e7439cbde13abaac72179d P 13089233e862406cb5dc1f847c76892a: stopping tablet replica
I20260812 05:22:01.689772 19360 raft_consensus.cc:2229] T ec886a8283e7439cbde13abaac72179d P 13089233e862406cb5dc1f847c76892a [term 1 LEADER]: Raft consensus shutting down.
I20260812 05:22:01.689893 19358 raft_consensus.cc:2229] T ec886a8283e7439cbde13abaac72179d P 7b308c61c8be4008a4442f3467143f70 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 05:22:01.689921 19360 raft_consensus.cc:2258] T ec886a8283e7439cbde13abaac72179d P 13089233e862406cb5dc1f847c76892a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 05:22:01.690168 19358 raft_consensus.cc:2258] T ec886a8283e7439cbde13abaac72179d P 7b308c61c8be4008a4442f3467143f70 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 05:22:01.692447 19358 ts_tablet_manager.cc:1905] T ec886a8283e7439cbde13abaac72179d P 7b308c61c8be4008a4442f3467143f70: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 05:22:01.693848 19360 ts_tablet_manager.cc:1905] T ec886a8283e7439cbde13abaac72179d P 13089233e862406cb5dc1f847c76892a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 05:22:01.694222 19358 ts_tablet_manager.cc:1918] T ec886a8283e7439cbde13abaac72179d P 7b308c61c8be4008a4442f3467143f70: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 05:22:01.694362 19358 log.cc:1196] T ec886a8283e7439cbde13abaac72179d P 7b308c61c8be4008a4442f3467143f70: Deleting WAL directory at /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786512017613366-32144-0/minicluster-data/ts-1/wal/wals/ec886a8283e7439cbde13abaac72179d
I20260812 05:22:01.694675 19358 ts_tablet_manager.cc:1939] T ec886a8283e7439cbde13abaac72179d P 7b308c61c8be4008a4442f3467143f70: Deleting consensus metadata
I20260812 05:22:01.695200 18396 catalog_manager.cc:4841] TS 7b308c61c8be4008a4442f3467143f70 (127.31.100.2:44677): tablet ec886a8283e7439cbde13abaac72179d (table test-table [id=214d70554e4b4c0fb0ec1123528979d4]) successfully deleted
I20260812 05:22:01.696463 19360 ts_tablet_manager.cc:1918] T ec886a8283e7439cbde13abaac72179d P 13089233e862406cb5dc1f847c76892a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 05:22:01.696552 19360 log.cc:1196] T ec886a8283e7439cbde13abaac72179d P 13089233e862406cb5dc1f847c76892a: Deleting WAL directory at /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786512017613366-32144-0/minicluster-data/ts-2/wal/wals/ec886a8283e7439cbde13abaac72179d
I20260812 05:22:01.696893 19360 ts_tablet_manager.cc:1939] T ec886a8283e7439cbde13abaac72179d P 13089233e862406cb5dc1f847c76892a: Deleting consensus metadata
I20260812 05:22:01.697340 18396 catalog_manager.cc:4841] TS 13089233e862406cb5dc1f847c76892a (127.31.100.3:37899): tablet ec886a8283e7439cbde13abaac72179d (table test-table [id=214d70554e4b4c0fb0ec1123528979d4]) successfully deleted
W20260812 05:22:01.702481 18396 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.31.100.1:34077 (error 108)
I20260812 05:22:01.703276 32144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0waFvT/build/release/bin/kudu with pid 18806
I20260812 05:22:01.706357 19313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:22:01.706202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.1:34077 (local address 127.0.0.1:60622)
0812 05:22:01.706257 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0812 05:22:01.706316 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.100.1:34077: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260812 05:22:01.706489 18396 catalog_manager.cc:4584] TS ffc155eed9214c8a80115ae4d04f4cbe (127.31.100.1:34077): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ec886a8283e7439cbde13abaac72179d: Network error: Client connection negotiation failed: client connection to 127.31.100.1:34077: connect: Connection refused (error 111)
I20260812 05:22:01.713038 32144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0waFvT/build/release/bin/kudu with pid 19059
I20260812 05:22:01.723374 32144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0waFvT/build/release/bin/kudu with pid 18369
I20260812 05:22:01.741277 32144 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260812 05:22:02.085036 32144 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260812 05:22:02.085147 32144 mini_ranger.cc:66] Stopping Ranger...
I20260812 05:22:02.428501 32144 mini_ranger.cc:68] Stopped Ranger
2026-08-12 05:22:02.428 UTC [17627] LOG: received smart shutdown request
2026-08-12 05:22:02.432 UTC [17627] LOG: background worker "logical replication launcher" (PID 17646) exited with exit code 1
2026-08-12 05:22:02.432 UTC [17637] LOG: shutting down
2026-08-12 05:22:02.576 UTC [17627] LOG: database system is shut down
2026-08-12T05:22:02Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (46482 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786512017613366-32144-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 05:22:02 dist-test-slave-42z9 krb5kdc[19378](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 05:22:02 dist-test-slave-42z9 krb5kdc[19378](info): set up 2 sockets
Aug 12 05:22:02 dist-test-slave-42z9 krb5kdc[19378](info): commencing operation
krb5kdc: starting...
W20260812 05:22:03.828994 32144 mini_kdc.cc:121] Time spent starting KDC: real 1.142s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786512017613366-32144-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786512017613366-32144-0/krb5kdc/test-user.keytab.
Aug 12 05:22:03 dist-test-slave-42z9 krb5kdc[19378](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512123, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T05:22:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T05:22:03Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.31.100.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.31.100.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.31.100.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786512017613366-32144-0/krb5kdc/rangeradmin_127.31.100.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.31.100.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786512017613366-32144-0/krb5kdc/rangeradmin_127.31.100.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.31.100.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.31.100.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.31.100.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786512017613366-32144-0/krb5kdc/rangerlookup_127.31.100.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.31.100.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786512017613366-32144-0/krb5kdc/rangerlookup_127.31.100.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.31.100.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.31.100.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.31.100.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786512017613366-32144-0/krb5kdc/HTTP_127.31.100.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.31.100.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786512017613366-32144-0/krb5kdc/HTTP_127.31.100.20@KRBTEST.COM.keytab.
I20260812 05:22:03.979826 32144 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786512017613366-32144-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-task0waFvT/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786512017613366-32144-0/minicluster-data/postgres -l logfile start
2026-08-12 05:22:06.260 UTC [19425] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-08-12 05:22:06.260 UTC [19425] LOG: listening on IPv4 address "127.31.100.20", port 60377
2026-08-12 05:22:06.263 UTC [19425] LOG: listening on Unix socket "/tmp/.s.PGSQL.60377"
2026-08-12 05:22:06.282 UTC [19428] LOG: database system was shut down at 2026-08-12 05:22:05 UTC
2026-08-12 05:22:06.285 UTC [19425] LOG: database system is ready to accept connections
I20260812 05:22:07.365061 32144 mini_postgres.cc:96] Postgres bound to 60377
2026-08-12 05:22:07.372 UTC [19444] FATAL: database "slave" does not exist
127.31.100.20:60377 - accepting connections
I20260812 05:22:07.372529 32144 mini_ranger.cc:161] Starting Ranger...
I20260812 05:22:07.387006 32144 mini_ranger.cc:84] Created miniranger Postgres user
I20260812 05:22:07.799261 32144 mini_ranger.cc:87] Created ranger Postgres database
I20260812 05:22:07.799333 32144 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786512017613366-32144-0/minicluster-data/ranger-admin
2026-08-12 05:22:08,060 [I] DB FLAVOR :POSTGRES
2026-08-12 05:22:08,060 [I] --------- Verifying Ranger DB connection ---------
2026-08-12 05:22:08,060 [I] Checking connection..
2026-08-12 05:22:08,060 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0waFvT/build/release/bin/postgresql.jar:/tmp/dist-test-task0waFvT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.100.20:60377/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-08-12 05:22:08,319 [I] Checking connection passed.
2026-08-12 05:22:08,320 [I] --------- Verifying version history table ---------
2026-08-12 05:22:08,320 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0waFvT/build/release/bin/postgresql.jar:/tmp/dist-test-task0waFvT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.100.20:60377/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-12 05:22:08,580 [I] Table x_db_version_h does not exist in database ranger
2026-08-12 05:22:08,580 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0waFvT/build/release/bin/postgresql.jar:/tmp/dist-test-task0waFvT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.100.20:60377/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-12 05:22:08,817 [I] Table x_db_version_h does not exist in database ranger
2026-08-12 05:22:08,817 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-08-12 05:22:08,817 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0waFvT/build/release/bin/postgresql.jar:/tmp/dist-test-task0waFvT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.100.20:60377/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task0waFvT/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-08-12 05:22:09.065 UTC [19557] WARNING: there is no transaction in progress
2026-08-12 05:22:09,074 [I] create_dbversion_catalog.sql file imported successfully
2026-08-12 05:22:09,074 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0waFvT/build/release/bin/postgresql.jar:/tmp/dist-test-task0waFvT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.100.20:60377/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-12 05:22:09,318 [I] Table x_db_version_h already exists in database 'ranger'
2026-08-12 05:22:09,319 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-12 05:22:09,319 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0waFvT/build/release/bin/postgresql.jar:/tmp/dist-test-task0waFvT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.100.20:60377/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-08-12 05:22:09,572 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0waFvT/build/release/bin/postgresql.jar:/tmp/dist-test-task0waFvT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.100.20:60377/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-08-12 05:22:09,816 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0waFvT/build/release/bin/postgresql.jar:/tmp/dist-test-task0waFvT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.100.20:60377/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-42z9.c.gcp-upstream.internal','N') ;"
2026-08-12 05:22:10,052 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0waFvT/build/release/bin/postgresql.jar:/tmp/dist-test-task0waFvT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.100.20:60377/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-12 05:22:10,317 [I] Table x_portal_user does not exist in database ranger
2026-08-12 05:22:10,317 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0waFvT/build/release/bin/postgresql.jar:/tmp/dist-test-task0waFvT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.100.20:60377/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-12 05:22:10,574 [I] Table x_policy_ref_group does not exist in database ranger
2026-08-12 05:22:10,575 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0waFvT/build/release/bin/postgresql.jar:/tmp/dist-test-task0waFvT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.100.20:60377/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-12 05:22:10,829 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0waFvT/build/release/bin/postgresql.jar:/tmp/dist-test-task0waFvT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.100.20:60377/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-12 05:22:11,080 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-08-12 05:22:11,080 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0waFvT/build/release/bin/postgresql.jar:/tmp/dist-test-task0waFvT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.100.20:60377/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task0waFvT/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-08-12 05:22:11.956 UTC [19740] WARNING: there is no transaction in progress
2026-08-12 05:22:11.970 UTC [19740] WARNING: there is no transaction in progress
2026-08-12 05:22:11.983 UTC [19740] WARNING: there is no transaction in progress
2026-08-12 05:22:11.997 UTC [19740] WARNING: there is no transaction in progress
2026-08-12 05:22:12.013 UTC [19740] WARNING: there is no transaction in progress
2026-08-12 05:22:12.029 UTC [19740] WARNING: there is no transaction in progress
2026-08-12 05:22:12.079 UTC [19740] WARNING: there is no transaction in progress
2026-08-12 05:22:12.086 UTC [19740] WARNING: there is no transaction in progress
2026-08-12 05:22:12.099 UTC [19740] WARNING: there is no transaction in progress
2026-08-12 05:22:12.108 UTC [19740] WARNING: there is no transaction in progress
2026-08-12 05:22:12.118 UTC [19740] WARNING: there is no transaction in progress
2026-08-12 05:22:12.131 UTC [19740] WARNING: there is no transaction in progress
2026-08-12 05:22:12.140 UTC [19740] WARNING: there is no transaction in progress
2026-08-12 05:22:12.146 UTC [19740] WARNING: there is no transaction in progress
2026-08-12 05:22:12.152 UTC [19740] WARNING: there is no transaction in progress
2026-08-12 05:22:12.157 UTC [19740] WARNING: there is no transaction in progress
2026-08-12 05:22:12,708 [I] ranger_core_db_postgres.sql file imported successfully
2026-08-12 05:22:12,708 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0waFvT/build/release/bin/postgresql.jar:/tmp/dist-test-task0waFvT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.100.20:60377/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-08-12 05:22:12,957 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-08-12 05:22:12,958 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0waFvT/build/release/bin/postgresql.jar:/tmp/dist-test-task0waFvT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.100.20:60377/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-12 05:22:13,200 [I] Table x_portal_user already exists in database 'ranger'
2026-08-12 05:22:13,201 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0waFvT/build/release/bin/postgresql.jar:/tmp/dist-test-task0waFvT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.100.20:60377/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-12 05:22:13,450 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-08-12 05:22:13,450 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0waFvT/build/release/bin/postgresql.jar:/tmp/dist-test-task0waFvT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.100.20:60377/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-12 05:22:13,704 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0waFvT/build/release/bin/postgresql.jar:/tmp/dist-test-task0waFvT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.100.20:60377/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-12 05:22:13,940 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0waFvT/build/release/bin/postgresql.jar:/tmp/dist-test-task0waFvT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.100.20:60377/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-42z9.c.gcp-upstream.internal';"
2026-08-12 05:22:14,163 [I] CORE_DB_SCHEMA import status has been updated
2026-08-12 05:22:14,163 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0waFvT/build/release/bin/postgresql.jar:/tmp/dist-test-task0waFvT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.100.20:60377/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-08-12 05:22:14,416 [I] DB_PATCHES have already been applied
I20260812 05:22:14.421846 32144 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786512017613366-32144-0/minicluster-data/ranger-admin:/tmp/dist-test-task0waFvT/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-task0waFvT/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-task0waFvT/build/release/bin/postgresql.jar:/tmp/dist-test-task0waFvT/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260812 05:22:14.421908 32144 mini_ranger.cc:193] Using host: 127.31.100.20
I20260812 05:22:14.425467 32144 mini_ranger.cc:227] Ranger admin URL: http://127.31.100.20:56229
Aug 12, 2026 5:22:14 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 12, 2026 5:22:14 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 12, 2026 5:22:14 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786512017613366-32144-0/minicluster-data/ranger-admin/ews/webapp
Aug 12, 2026 5:22:14 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786512017613366-32144-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Aug 12, 2026 5:22:14 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786512017613366-32144-0/minicluster-data/ranger-admin/ews/webapp] .....
Aug 12, 2026 5:22:15 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786512017613366-32144-0/minicluster-data/ranger-admin/ews/webapp].
Aug 12, 2026 5:22:15 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260812 05:22:15.643959 32144 mini_ranger.cc:160] Time spent starting Ranger: real 8.271s user 0.003s sys 0.006s
Aug 12, 2026 5:22:35 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Aug 12, 2026 5:22:35 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Aug 12, 2026 5:22:35 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Aug 12, 2026 5:22:35 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Aug 12, 2026 5:22:35 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Aug 12, 2026 5:22:35 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Aug 12, 2026 5:22:35 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Aug 12, 2026 5:22:35 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Aug 12, 2026 5:22:35 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Aug 12, 2026 5:22:35 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Aug 12, 2026 5:22:35 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Aug 12, 2026 5:22:35 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Aug 12, 2026 5:22:35 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Aug 12, 2026 5:22:35 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Aug 12, 2026 5:22:35 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Aug 12, 2026 5:22:35 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Aug 12, 2026 5:22:35 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Aug 12, 2026 5:22:36 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260812 05:22:36.730631 32144 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.31.100.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.31.100.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.31.100.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786512017613366-32144-0/krb5kdc/rangerkms_127.31.100.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.31.100.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786512017613366-32144-0/krb5kdc/rangerkms_127.31.100.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.31.100.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.31.100.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.31.100.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786512017613366-32144-0/krb5kdc/HTTP_127.31.100.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.31.100.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786512017613366-32144-0/krb5kdc/HTTP_127.31.100.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Aug 12 05:22:36 dist-test-slave-42z9 krb5kdc[19378](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512156, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260812 05:22:36.810122 32144 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260812 05:22:36.819525 32144 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260812 05:22:37.671031 32144 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260812 05:22:37.671116 32144 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786512017613366-32144-0/minicluster-data/ranger-kms
I20260812 05:22:37.671130 32144 mini_ranger_kms.cc:227] Using postgres at 127.31.100.20:60377
2026-08-12 05:22:37,828 [I] DB FLAVOR :POSTGRES
2026-08-12 05:22:37,828 [I] --------- Verifying Ranger DB connection ---------
2026-08-12 05:22:37,828 [I] Checking connection
2026-08-12 05:22:37,828 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0waFvT/build/release/bin/postgresql.jar:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786512017613366-32144-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.100.20:60377/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-08-12 05:22:38,073 [I] connection success
2026-08-12 05:22:38,073 [I] --------- Verifying Ranger DB tables ---------
2026-08-12 05:22:38,073 [I] Verifying table ranger_masterkey in database rangerkms
2026-08-12 05:22:38,073 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0waFvT/build/release/bin/postgresql.jar:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786512017613366-32144-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.100.20:60377/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-08-12 05:22:38,327 [I] Table ranger_masterkey does not exist in database rangerkms
2026-08-12 05:22:38,328 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-12 05:22:38,328 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-08-12 05:22:38,328 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0waFvT/build/release/bin/postgresql.jar:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786512017613366-32144-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.100.20:60377/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786512017613366-32144-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-08-12 05:22:38,580 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260812 05:22:39.021687 32144 mini_ranger_kms.cc:310] Created kms service
I20260812 05:22:39.216235 32144 mini_ranger_kms.cc:326] Created kudu user
I20260812 05:22:39.261529 32144 mini_ranger_kms.cc:343] Created rangerkms user
Aug 12 05:22:39 dist-test-slave-42z9 krb5kdc[19378](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512156, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.31.100.20@KRBTEST.COM
I20260812 05:22:39.633279 32144 mini_ranger_kms.cc:384] Added ranger policy
I20260812 05:22:39.633540 32144 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786512017613366-32144-0/minicluster-data/ranger-kms:/tmp/dist-test-task0waFvT/build/release/bin/postgresql.jar:/tmp/dist-test-task0waFvT/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-task0waFvT/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260812 05:22:39.633572 32144 mini_ranger_kms.cc:240] Using host: 127.31.100.20
I20260812 05:22:39.638294 32144 mini_ranger_kms.cc:276] Ranger KMS PID: 20050
I20260812 05:22:39.638360 32144 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.31.100.20:56645
Aug 12, 2026 5:22:40 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 12, 2026 5:22:40 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 12, 2026 5:22:40 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786512017613366-32144-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Aug 12, 2026 5:22:40 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786512017613366-32144-0/minicluster-data/ranger-kms/ews/webapp] .....
Aug 12, 2026 5:22:41 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786512017613366-32144-0/minicluster-data/ranger-kms/ews/webapp].
Aug 12, 2026 5:22:41 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Aug 12 05:22:41 dist-test-slave-42z9 krb5kdc[19378](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786512161, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.31.100.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 05:22:42.187067 32144 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.377s user 0.003s sys 0.139s
I20260812 05:22:42.187175 32144 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260812 05:22:42.187222 32144 mini_ranger_kms.cc:401] 127.31.100.20:56645/kms/v1/keys
Aug 12 05:22:42 dist-test-slave-42z9 krb5kdc[19378](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512161, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.31.100.20@KRBTEST.COM for HTTP/127.31.100.20@KRBTEST.COM
Aug 12 05:22:42 dist-test-slave-42z9 krb5kdc[19378](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512161, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.31.100.20@KRBTEST.COM for HTTP/127.31.100.20@KRBTEST.COM
Aug 12 05:22:44 dist-test-slave-42z9 krb5kdc[19378](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512164, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.31.100.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.100.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.100.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.62.keytab.
Entry for principal kudu/127.31.100.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.62.keytab.
I20260812 05:22:44.836089 32144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0waFvT/build/release/bin/kudu
/tmp/dist-test-task0waFvT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786512017613366-32144-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786512017613366-32144-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786512017613366-32144-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786512017613366-32144-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.100.62:42701
--webserver_interface=127.31.100.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.62.keytab
--principal=kudu/127.31.100.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.100.20:44553
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.100.62:42701
--ranger_config_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786512017613366-32144-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.31.100.20:56645/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786512017613366-32144-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786512017613366-32144-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786512017613366-32144-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 05:22:44.916054 20135 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 05:22:44.916237 20135 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 05:22:44.916257 20135 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 05:22:44.917385 20135 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 05:22:44.917430 20135 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 05:22:44.917472 20135 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 05:22:44.917490 20135 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 05:22:44.917505 20135 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 05:22:44.919005 20135 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.100.20:44553
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786512017613366-32144-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786512017613366-32144-0/minicluster-data/master-0/wal
--ranger_kms_url=127.31.100.20:56645/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.31.100.62:42701
--ranger_config_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786512017613366-32144-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.62.keytab
--principal=kudu/127.31.100.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.100.62:42701
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786512017613366-32144-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.100.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20135
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786512017613366-32144-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 04:20:34 UTC on ccycloud.quasar-phqdqp.root.comops.site
build id 4581871
I20260812 05:22:44.919214 20135 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 05:22:44.919435 20135 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260812 05:22:44.924620 20135 server_base.cc:1034] running on GCE node
W20260812 05:22:44.924633 20145 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 05:22:44.924736 20146 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 05:22:44.924780 20148 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 05:22:44.925041 20135 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 05:22:44.925369 20135 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 05:22:44.926544 20135 hybrid_clock.cc:648] HybridClock initialized: now 1786512164926536 us; error 47 us; skew 500 ppm
Aug 12 05:22:44 dist-test-slave-42z9 krb5kdc[19378](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512164, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.100.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 05:22:44.929630 20135 init.cc:377] Logged in from keytab as kudu/127.31.100.62@KRBTEST.COM (short username kudu)
I20260812 05:22:44.930184 20135 webserver.cc:460] Webserver started at http://127.31.100.62:37371/ using document root <none> and password file <none>
I20260812 05:22:44.930426 20135 fs_manager.cc:359] Metadata directory not provided
I20260812 05:22:44.930490 20135 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 05:22:44.930629 20135 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 12 05:22:44 dist-test-slave-42z9 krb5kdc[19378](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512164, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.100.62@KRBTEST.COM for HTTP/127.31.100.20@KRBTEST.COM
I20260812 05:22:44.982070 20135 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786512017613366-32144-0/minicluster-data/master-0/data/instance:
uuid: "04405e5fcf124400abdb1c71ecfb4985"
format_stamp: "Formatted at 2026-08-12 05:22:44 on dist-test-slave-42z9"
server_key: "7d7f25bd8e55ccb891b4a6c6d6d47073"
server_key_iv: "8e9d96534a9f3b0c32935941fb130987"
server_key_version: "kuduclusterkey@0"
I20260812 05:22:44.982522 20135 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786512017613366-32144-0/minicluster-data/master-0/wal/instance:
uuid: "04405e5fcf124400abdb1c71ecfb4985"
format_stamp: "Formatted at 2026-08-12 05:22:44 on dist-test-slave-42z9"
server_key: "7d7f25bd8e55ccb891b4a6c6d6d47073"
server_key_iv: "8e9d96534a9f3b0c32935941fb130987"
server_key_version: "kuduclusterkey@0"
I20260812 05:22:44.984014 20135 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260812 05:22:44.999104 20162 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 05:22:44.999321 20135 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260812 05:22:44.999431 20135 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786512017613366-32144-0/minicluster-data/master-0/data,/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786512017613366-32144-0/minicluster-data/master-0/wal
uuid: "04405e5fcf124400abdb1c71ecfb4985"
format_stamp: "Formatted at 2026-08-12 05:22:44 on dist-test-slave-42z9"
server_key: "7d7f25bd8e55ccb891b4a6c6d6d47073"
server_key_iv: "8e9d96534a9f3b0c32935941fb130987"
server_key_version: "kuduclusterkey@0"
I20260812 05:22:44.999500 20135 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786512017613366-32144-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786512017613366-32144-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786512017613366-32144-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 05:22:45.014550 20135 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 05:22:45.016427 20135 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 05:22:45.016566 20135 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 05:22:45.021025 20135 rpc_server.cc:307] RPC server started. Bound to: 127.31.100.62:42701
I20260812 05:22:45.021049 20256 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.100.62:42701 every 8 connection(s)
I20260812 05:22:45.021467 20135 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786512017613366-32144-0/minicluster-data/master-0/data/info.pb
I20260812 05:22:45.021621 32144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0waFvT/build/release/bin/kudu as pid 20135
I20260812 05:22:45.021723 32144 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786512017613366-32144-0/minicluster-data/master-0/wal/instance
I20260812 05:22:45.021930 32144 external_mini_cluster.cc:1428] Setting key 57550f97a47fe692bb9e8cecfcfe5a59
I20260812 05:22:45.022367 20258 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 05:22:45.025471 20258 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 04405e5fcf124400abdb1c71ecfb4985: Bootstrap starting.
I20260812 05:22:45.026067 20258 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 04405e5fcf124400abdb1c71ecfb4985: Neither blocks nor log segments found. Creating new log.
I20260812 05:22:45.026463 20258 log.cc:824] T 00000000000000000000000000000000 P 04405e5fcf124400abdb1c71ecfb4985: Log is configured to *not* fsync() on all Append() calls
I20260812 05:22:45.027300 20258 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 04405e5fcf124400abdb1c71ecfb4985: No bootstrap required, opened a new log
Aug 12 05:22:45 dist-test-slave-42z9 krb5kdc[19378](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512164, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.100.62@KRBTEST.COM
I20260812 05:22:45.029759 20258 raft_consensus.cc:348] T 00000000000000000000000000000000 P 04405e5fcf124400abdb1c71ecfb4985 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04405e5fcf124400abdb1c71ecfb4985" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 42701 } }
I20260812 05:22:45.029944 20258 raft_consensus.cc:374] T 00000000000000000000000000000000 P 04405e5fcf124400abdb1c71ecfb4985 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 05:22:45.030010 20258 raft_consensus.cc:729] T 00000000000000000000000000000000 P 04405e5fcf124400abdb1c71ecfb4985 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04405e5fcf124400abdb1c71ecfb4985, State: Initialized, Role: FOLLOWER
I20260812 05:22:45.030232 20258 consensus_queue.cc:260] T 00000000000000000000000000000000 P 04405e5fcf124400abdb1c71ecfb4985 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04405e5fcf124400abdb1c71ecfb4985" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 42701 } }
I20260812 05:22:45.030315 20258 raft_consensus.cc:388] T 00000000000000000000000000000000 P 04405e5fcf124400abdb1c71ecfb4985 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 05:22:45.030344 20258 raft_consensus.cc:482] T 00000000000000000000000000000000 P 04405e5fcf124400abdb1c71ecfb4985 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 05:22:45.030408 20258 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 04405e5fcf124400abdb1c71ecfb4985 [term 0 FOLLOWER]: Advancing to term 1
I20260812 05:22:45.031076 20258 raft_consensus.cc:504] T 00000000000000000000000000000000 P 04405e5fcf124400abdb1c71ecfb4985 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04405e5fcf124400abdb1c71ecfb4985" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 42701 } }
I20260812 05:22:45.031190 20258 leader_election.cc:302] T 00000000000000000000000000000000 P 04405e5fcf124400abdb1c71ecfb4985 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 04405e5fcf124400abdb1c71ecfb4985; no voters:
I20260812 05:22:45.031389 20258 leader_election.cc:288] T 00000000000000000000000000000000 P 04405e5fcf124400abdb1c71ecfb4985 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 05:22:45.031679 20258 sys_catalog.cc:564] T 00000000000000000000000000000000 P 04405e5fcf124400abdb1c71ecfb4985 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 05:22:45.032819 20261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:22:45.022415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57920 (local address 127.31.100.62:42701)
0812 05:22:45.022687 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0812 05:22:45.022698 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0812 05:22:45.022736 (+ 38us) server_negotiation.cc:407] Connection header received
0812 05:22:45.023527 (+ 791us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:22:45.023537 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:22:45.023601 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:22:45.023750 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:22:45.024530 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:22:45.025974 (+ 1444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:22:45.026976 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:22:45.027383 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:22:45.027490 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:22:45.029702 (+ 2212us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 05:22:45.029707 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 05:22:45.029707 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 05:22:45.029719 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0812 05:22:45.031244 (+ 1525us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:22:45.031613 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:22:45.031616 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:22:45.031616 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:22:45.031701 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:22:45.031896 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:22:45.031897 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:22:45.031898 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:22:45.032143 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 05:22:45.032289 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0812 05:22:45.032517 (+ 228us) server_negotiation.cc:299] Negotiation successful
0812 05:22:45.032630 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":221,"thread_start_us":75,"threads_started":1}
I20260812 05:22:45.033187 20258 ranger_client.cc:318] Using new properties file: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786512017613366-32144-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260812 05:22:45.035523 20263 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 04405e5fcf124400abdb1c71ecfb4985 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 05:22:45.035985 20263 raft_consensus.cc:686] T 00000000000000000000000000000000 P 04405e5fcf124400abdb1c71ecfb4985 [term 1 LEADER]: Becoming Leader. State: Replica: 04405e5fcf124400abdb1c71ecfb4985, State: Running, Role: LEADER
I20260812 05:22:45.036154 20263 consensus_queue.cc:237] T 00000000000000000000000000000000 P 04405e5fcf124400abdb1c71ecfb4985 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04405e5fcf124400abdb1c71ecfb4985" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 42701 } }
I20260812 05:22:45.036567 20263 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04405e5fcf124400abdb1c71ecfb4985 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 04405e5fcf124400abdb1c71ecfb4985. Latest consensus state: current_term: 1 leader_uuid: "04405e5fcf124400abdb1c71ecfb4985" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04405e5fcf124400abdb1c71ecfb4985" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 42701 } } }
I20260812 05:22:45.036580 20264 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04405e5fcf124400abdb1c71ecfb4985 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "04405e5fcf124400abdb1c71ecfb4985" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04405e5fcf124400abdb1c71ecfb4985" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 42701 } } }
I20260812 05:22:45.036677 20263 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04405e5fcf124400abdb1c71ecfb4985 [sys.catalog]: This master's current role is: LEADER
I20260812 05:22:45.036710 20264 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04405e5fcf124400abdb1c71ecfb4985 [sys.catalog]: This master's current role is: LEADER
I20260812 05:22:45.041407 20285 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 05:22:45.041908 20285 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 05:22:45.045818 20285 catalog_manager.cc:1347] Generated new cluster ID: adfb6b5d097d44f292e52a24fc9c8dce
I20260812 05:22:45.045874 20285 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 05:22:45.057796 20285 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 05:22:45.058566 20285 catalog_manager.cc:1504] Loading token signing keys...
I20260812 05:22:45.064870 20285 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 04405e5fcf124400abdb1c71ecfb4985: Generated new TSK 0
I20260812 05:22:45.065150 20285 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.31.100.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.100.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.100.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.1.keytab.
Entry for principal kudu/127.31.100.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.1.keytab.
I20260812 05:22:45.784723 32144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0waFvT/build/release/bin/kudu
/tmp/dist-test-task0waFvT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786512017613366-32144-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786512017613366-32144-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786512017613366-32144-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786512017613366-32144-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.31.100.20:56645/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.100.1:0
--local_ip_for_outbound_sockets=127.31.100.1
--webserver_interface=127.31.100.1
--webserver_port=0
--tserver_master_addrs=127.31.100.62:42701
--keytab_file=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.1.keytab
--principal=kudu/127.31.100.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.100.20:44553
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786512017613366-32144-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786512017613366-32144-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786512017613366-32144-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 05:22:45.894695 20334 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 05:22:45.894896 20334 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 05:22:45.894918 20334 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 05:22:45.896027 20334 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 05:22:45.896065 20334 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 05:22:45.896090 20334 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.100.1
I20260812 05:22:45.897537 20334 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.100.20:44553
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786512017613366-32144-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786512017613366-32144-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.31.100.20:56645/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.1.keytab
--principal=kudu/127.31.100.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.100.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786512017613366-32144-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.100.1
--webserver_port=0
--tserver_master_addrs=127.31.100.62:42701
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20334
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.100.1
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786512017613366-32144-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 04:20:34 UTC on ccycloud.quasar-phqdqp.root.comops.site
build id 4581871
I20260812 05:22:45.897732 20334 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 05:22:45.897967 20334 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 05:22:45.902985 20345 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 05:22:45.903014 20348 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 05:22:45.903231 20351 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 05:22:45.903441 20334 server_base.cc:1034] running on GCE node
I20260812 05:22:45.903687 20334 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 05:22:45.903930 20334 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 05:22:45.905079 20334 hybrid_clock.cc:648] HybridClock initialized: now 1786512165905064 us; error 41 us; skew 500 ppm
Aug 12 05:22:45 dist-test-slave-42z9 krb5kdc[19378](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512165, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.100.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 05:22:45.907488 20334 init.cc:377] Logged in from keytab as kudu/127.31.100.1@KRBTEST.COM (short username kudu)
I20260812 05:22:45.907883 20334 webserver.cc:460] Webserver started at http://127.31.100.1:42803/ using document root <none> and password file <none>
I20260812 05:22:45.908094 20334 fs_manager.cc:359] Metadata directory not provided
I20260812 05:22:45.908144 20334 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 05:22:45.908263 20334 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 12 05:22:45 dist-test-slave-42z9 krb5kdc[19378](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512165, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.100.1@KRBTEST.COM for HTTP/127.31.100.20@KRBTEST.COM
I20260812 05:22:45.923681 20334 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786512017613366-32144-0/minicluster-data/ts-0/data/instance:
uuid: "0502228f6e7340dea89eeaf2aa9f6e75"
format_stamp: "Formatted at 2026-08-12 05:22:45 on dist-test-slave-42z9"
server_key: "ba716f95e597fe7de1d7c4c125b2583e"
server_key_iv: "5a7f079e618cdb9afd0b009e84e57193"
server_key_version: "kuduclusterkey@0"
I20260812 05:22:45.924157 20334 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786512017613366-32144-0/minicluster-data/ts-0/wal/instance:
uuid: "0502228f6e7340dea89eeaf2aa9f6e75"
format_stamp: "Formatted at 2026-08-12 05:22:45 on dist-test-slave-42z9"
server_key: "ba716f95e597fe7de1d7c4c125b2583e"
server_key_iv: "5a7f079e618cdb9afd0b009e84e57193"
server_key_version: "kuduclusterkey@0"
I20260812 05:22:45.930071 20334 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.002s
I20260812 05:22:45.944453 20364 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 05:22:45.944669 20334 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260812 05:22:45.944770 20334 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786512017613366-32144-0/minicluster-data/ts-0/data,/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786512017613366-32144-0/minicluster-data/ts-0/wal
uuid: "0502228f6e7340dea89eeaf2aa9f6e75"
format_stamp: "Formatted at 2026-08-12 05:22:45 on dist-test-slave-42z9"
server_key: "ba716f95e597fe7de1d7c4c125b2583e"
server_key_iv: "5a7f079e618cdb9afd0b009e84e57193"
server_key_version: "kuduclusterkey@0"
I20260812 05:22:45.944859 20334 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786512017613366-32144-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786512017613366-32144-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786512017613366-32144-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 05:22:45.969636 20334 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 05:22:45.971899 20334 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 05:22:45.972044 20334 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 05:22:45.972301 20334 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 05:22:45.972702 20334 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 05:22:45.972790 20334 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 05:22:45.972860 20334 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 05:22:45.972909 20334 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 05:22:45.981412 20334 rpc_server.cc:307] RPC server started. Bound to: 127.31.100.1:33025
I20260812 05:22:45.981536 20550 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.100.1:33025 every 8 connection(s)
I20260812 05:22:45.981894 20334 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786512017613366-32144-0/minicluster-data/ts-0/data/info.pb
I20260812 05:22:45.982981 32144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0waFvT/build/release/bin/kudu as pid 20334
I20260812 05:22:45.983083 32144 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786512017613366-32144-0/minicluster-data/ts-0/wal/instance
I20260812 05:22:45.983250 32144 external_mini_cluster.cc:1428] Setting key 905b45bfcfbdd457cbfdeeeb0f987214
WARNING: no policy specified for kudu/127.31.100.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.100.2@KRBTEST.COM" created.
Aug 12 05:22:45 dist-test-slave-42z9 krb5kdc[19378](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512165, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.100.1@KRBTEST.COM for kudu/127.31.100.62@KRBTEST.COM
I20260812 05:22:46.003832 20555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:22:45.982738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.100.1:60983 (local address 127.31.100.62:42701)
0812 05:22:45.982947 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0812 05:22:45.982949 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 05:22:45.983544 (+ 595us) server_negotiation.cc:407] Connection header received
0812 05:22:45.986236 (+ 2692us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:22:45.986237 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:22:45.986251 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:22:45.986297 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:22:45.993380 (+ 7083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:22:45.993979 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:22:45.995097 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:22:45.995395 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:22:45.995491 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:22:46.000014 (+ 4523us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 05:22:46.000016 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 05:22:46.000017 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 05:22:46.000026 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0812 05:22:46.002127 (+ 2101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:22:46.002644 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:22:46.002646 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:22:46.002646 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:22:46.002718 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:22:46.003031 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:22:46.003032 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:22:46.003033 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:22:46.003174 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 05:22:46.003243 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0812 05:22:46.003631 (+ 388us) server_negotiation.cc:299] Negotiation successful
0812 05:22:46.003722 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":167,"thread_start_us":135,"threads_started":1}
I20260812 05:22:46.004033 20553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:22:45.982655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.62:42701 (local address 127.31.100.1:60983)
0812 05:22:45.983414 (+ 759us) negotiation.cc:107] Waiting for socket to connect
0812 05:22:45.983473 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0812 05:22:45.986113 (+ 2640us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:22:45.986496 (+ 383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:22:45.986501 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:22:45.986855 (+ 354us) client_negotiation.cc:189] Negotiated authn=SASL
0812 05:22:45.987256 (+ 401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:22:45.987263 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:22:45.994124 (+ 6861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:22:45.994125 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:22:45.995010 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:22:45.995013 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:22:45.995521 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:22:45.995522 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:22:45.995823 (+ 301us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:22:45.997856 (+ 2033us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 05:22:45.997867 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0812 05:22:45.999891 (+ 2024us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 05:22:46.002216 (+ 2325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 05:22:46.002223 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 05:22:46.002226 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 05:22:46.002546 (+ 320us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 05:22:46.002793 (+ 247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 05:22:46.002794 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 05:22:46.002794 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 05:22:46.002940 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 05:22:46.003247 (+ 307us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 05:22:46.003255 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 05:22:46.003435 (+ 180us) client_negotiation.cc:769] Sending connection context
0812 05:22:46.003587 (+ 152us) client_negotiation.cc:240] Negotiation successful
0812 05:22:46.003760 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":72,"threads_started":1}
I20260812 05:22:46.004776 20551 heartbeater.cc:347] Connected to a master server at 127.31.100.62:42701
I20260812 05:22:46.004874 20551 heartbeater.cc:464] Registering TS with master...
I20260812 05:22:46.005082 20551 heartbeater.cc:511] Master 127.31.100.62:42701 requested a full tablet report, sending...
I20260812 05:22:46.005669 20192 ts_manager.cc:194] Registered new tserver with Master: 0502228f6e7340dea89eeaf2aa9f6e75 (127.31.100.1:33025)
I20260812 05:22:46.006354 20192 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.100.1@KRBTEST.COM'} at 127.31.100.1:60983
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.100.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.2.keytab.
Entry for principal kudu/127.31.100.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.2.keytab.
I20260812 05:22:46.011898 32144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0waFvT/build/release/bin/kudu
/tmp/dist-test-task0waFvT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786512017613366-32144-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786512017613366-32144-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786512017613366-32144-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786512017613366-32144-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.31.100.20:56645/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.100.2:0
--local_ip_for_outbound_sockets=127.31.100.2
--webserver_interface=127.31.100.2
--webserver_port=0
--tserver_master_addrs=127.31.100.62:42701
--keytab_file=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.2.keytab
--principal=kudu/127.31.100.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.100.20:44553
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786512017613366-32144-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786512017613366-32144-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786512017613366-32144-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 05:22:46.134862 20564 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 05:22:46.135192 20564 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 05:22:46.135286 20564 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 05:22:46.136920 20564 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 05:22:46.137029 20564 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 05:22:46.137120 20564 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.100.2
I20260812 05:22:46.138909 20564 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.100.20:44553
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786512017613366-32144-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786512017613366-32144-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.31.100.20:56645/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.2.keytab
--principal=kudu/127.31.100.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.100.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786512017613366-32144-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.100.2
--webserver_port=0
--tserver_master_addrs=127.31.100.62:42701
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20564
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.100.2
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786512017613366-32144-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 04:20:34 UTC on ccycloud.quasar-phqdqp.root.comops.site
build id 4581871
I20260812 05:22:46.139146 20564 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 05:22:46.139417 20564 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 05:22:46.145063 20576 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 05:22:46.145207 20575 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 05:22:46.145071 20581 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 05:22:46.146221 20564 server_base.cc:1034] running on GCE node
I20260812 05:22:46.146423 20564 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 05:22:46.146649 20564 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 05:22:46.147802 20564 hybrid_clock.cc:648] HybridClock initialized: now 1786512166147781 us; error 36 us; skew 500 ppm
Aug 12 05:22:46 dist-test-slave-42z9 krb5kdc[19378](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512166, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.100.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 05:22:46.150763 20564 init.cc:377] Logged in from keytab as kudu/127.31.100.2@KRBTEST.COM (short username kudu)
I20260812 05:22:46.151255 20564 webserver.cc:460] Webserver started at http://127.31.100.2:36851/ using document root <none> and password file <none>
I20260812 05:22:46.151500 20564 fs_manager.cc:359] Metadata directory not provided
I20260812 05:22:46.151537 20564 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 05:22:46.151625 20564 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 12 05:22:46 dist-test-slave-42z9 krb5kdc[19378](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512166, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.100.2@KRBTEST.COM for HTTP/127.31.100.20@KRBTEST.COM
I20260812 05:22:46.168071 20564 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786512017613366-32144-0/minicluster-data/ts-1/data/instance:
uuid: "d80067ce5a344f35bf1996cf0f6e20c8"
format_stamp: "Formatted at 2026-08-12 05:22:46 on dist-test-slave-42z9"
server_key: "da654db2ff57b498a6b4c70e12d57584"
server_key_iv: "a04f9977bda44dd72b246644aafdd0f8"
server_key_version: "kuduclusterkey@0"
I20260812 05:22:46.168712 20564 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786512017613366-32144-0/minicluster-data/ts-1/wal/instance:
uuid: "d80067ce5a344f35bf1996cf0f6e20c8"
format_stamp: "Formatted at 2026-08-12 05:22:46 on dist-test-slave-42z9"
server_key: "da654db2ff57b498a6b4c70e12d57584"
server_key_iv: "a04f9977bda44dd72b246644aafdd0f8"
server_key_version: "kuduclusterkey@0"
Aug 12 05:22:46 dist-test-slave-42z9 krb5kdc[19378](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786512166, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.100.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 05:22:46.171727 20564 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260812 05:22:46.197613 20595 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 05:22:46.197875 20564 fs_manager.cc:725] Time spent opening block manager: real 0.009s user 0.000s sys 0.001s
I20260812 05:22:46.198029 20564 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786512017613366-32144-0/minicluster-data/ts-1/data,/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786512017613366-32144-0/minicluster-data/ts-1/wal
uuid: "d80067ce5a344f35bf1996cf0f6e20c8"
format_stamp: "Formatted at 2026-08-12 05:22:46 on dist-test-slave-42z9"
server_key: "da654db2ff57b498a6b4c70e12d57584"
server_key_iv: "a04f9977bda44dd72b246644aafdd0f8"
server_key_version: "kuduclusterkey@0"
I20260812 05:22:46.198189 20564 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786512017613366-32144-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786512017613366-32144-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786512017613366-32144-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 05:22:46.232620 20564 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 05:22:46.235108 20564 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 05:22:46.235266 20564 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 05:22:46.235666 20564 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 05:22:46.236125 20564 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 05:22:46.236179 20564 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 05:22:46.236265 20564 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 05:22:46.236296 20564 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 05:22:46.248131 20564 rpc_server.cc:307] RPC server started. Bound to: 127.31.100.2:39845
I20260812 05:22:46.248674 20564 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786512017613366-32144-0/minicluster-data/ts-1/data/info.pb
I20260812 05:22:46.252137 20791 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.100.2:39845 every 8 connection(s)
I20260812 05:22:46.252831 32144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0waFvT/build/release/bin/kudu as pid 20564
I20260812 05:22:46.252945 32144 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786512017613366-32144-0/minicluster-data/ts-1/wal/instance
I20260812 05:22:46.253173 32144 external_mini_cluster.cc:1428] Setting key f04f6798d57d9eb28c9eed2438ff5fae
Aug 12 05:22:46 dist-test-slave-42z9 krb5kdc[19378](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512166, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.100.2@KRBTEST.COM for kudu/127.31.100.62@KRBTEST.COM
I20260812 05:22:46.270330 20799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:22:46.252843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.62:42701 (local address 127.31.100.2:59533)
0812 05:22:46.259431 (+ 6588us) negotiation.cc:107] Waiting for socket to connect
0812 05:22:46.259491 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0812 05:22:46.260384 (+ 893us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:22:46.260731 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:22:46.260736 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:22:46.261003 (+ 267us) client_negotiation.cc:189] Negotiated authn=SASL
0812 05:22:46.261303 (+ 300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:22:46.261310 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:22:46.262329 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:22:46.262330 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:22:46.263612 (+ 1282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:22:46.263615 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:22:46.264192 (+ 577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:22:46.264193 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:22:46.264334 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:22:46.264912 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 05:22:46.264923 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0812 05:22:46.266624 (+ 1701us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 05:22:46.268582 (+ 1958us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 05:22:46.268590 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 05:22:46.268592 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 05:22:46.268896 (+ 304us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 05:22:46.269130 (+ 234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 05:22:46.269131 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 05:22:46.269131 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 05:22:46.269270 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 05:22:46.269612 (+ 342us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 05:22:46.269619 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 05:22:46.269789 (+ 170us) client_negotiation.cc:769] Sending connection context
0812 05:22:46.269902 (+ 113us) client_negotiation.cc:240] Negotiation successful
0812 05:22:46.270075 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6527,"thread_start_us":88,"threads_started":1}
I20260812 05:22:46.271070 20555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:22:46.253148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.100.2:59533 (local address 127.31.100.62:42701)
0812 05:22:46.253195 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0812 05:22:46.253197 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 05:22:46.259587 (+ 6390us) server_negotiation.cc:407] Connection header received
0812 05:22:46.260518 (+ 931us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:22:46.260519 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:22:46.260534 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:22:46.260593 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:22:46.261423 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:22:46.262213 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:22:46.263718 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:22:46.264014 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:22:46.264079 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:22:46.266732 (+ 2653us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 05:22:46.266733 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 05:22:46.266733 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 05:22:46.266741 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0812 05:22:46.268486 (+ 1745us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:22:46.268984 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:22:46.268986 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:22:46.268986 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:22:46.269060 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:22:46.269349 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:22:46.269350 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:22:46.269350 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:22:46.269517 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 05:22:46.269583 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0812 05:22:46.270863 (+ 1280us) server_negotiation.cc:299] Negotiation successful
0812 05:22:46.270960 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260812 05:22:46.271417 20793 heartbeater.cc:347] Connected to a master server at 127.31.100.62:42701
I20260812 05:22:46.271510 20793 heartbeater.cc:464] Registering TS with master...
I20260812 05:22:46.271694 20793 heartbeater.cc:511] Master 127.31.100.62:42701 requested a full tablet report, sending...
I20260812 05:22:46.272303 20192 ts_manager.cc:194] Registered new tserver with Master: d80067ce5a344f35bf1996cf0f6e20c8 (127.31.100.2:39845)
I20260812 05:22:46.272782 20192 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.100.2@KRBTEST.COM'} at 127.31.100.2:59533
WARNING: no policy specified for kudu/127.31.100.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.100.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.100.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.3.keytab.
Entry for principal kudu/127.31.100.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.3.keytab.
I20260812 05:22:46.299573 32144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0waFvT/build/release/bin/kudu
/tmp/dist-test-task0waFvT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786512017613366-32144-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786512017613366-32144-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786512017613366-32144-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786512017613366-32144-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.31.100.20:56645/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.100.3:0
--local_ip_for_outbound_sockets=127.31.100.3
--webserver_interface=127.31.100.3
--webserver_port=0
--tserver_master_addrs=127.31.100.62:42701
--keytab_file=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.3.keytab
--principal=kudu/127.31.100.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.100.20:44553
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786512017613366-32144-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786512017613366-32144-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786512017613366-32144-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 05:22:46.433624 20805 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 05:22:46.433967 20805 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 05:22:46.434051 20805 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 05:22:46.435734 20805 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 05:22:46.435845 20805 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 05:22:46.435937 20805 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.100.3
I20260812 05:22:46.438222 20805 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.100.20:44553
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786512017613366-32144-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786512017613366-32144-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.31.100.20:56645/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.3.keytab
--principal=kudu/127.31.100.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.100.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786512017613366-32144-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.100.3
--webserver_port=0
--tserver_master_addrs=127.31.100.62:42701
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20805
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.100.3
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786512017613366-32144-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 04:20:34 UTC on ccycloud.quasar-phqdqp.root.comops.site
build id 4581871
I20260812 05:22:46.438656 20805 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 05:22:46.438958 20805 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 05:22:46.449003 20817 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 05:22:46.449034 20810 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 05:22:46.449425 20820 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 05:22:46.450385 20805 server_base.cc:1034] running on GCE node
I20260812 05:22:46.450683 20805 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 05:22:46.450943 20805 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 05:22:46.452096 20805 hybrid_clock.cc:648] HybridClock initialized: now 1786512166452071 us; error 36 us; skew 500 ppm
Aug 12 05:22:46 dist-test-slave-42z9 krb5kdc[19378](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512166, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.100.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 05:22:46.455505 20805 init.cc:377] Logged in from keytab as kudu/127.31.100.3@KRBTEST.COM (short username kudu)
I20260812 05:22:46.456000 20805 webserver.cc:460] Webserver started at http://127.31.100.3:40811/ using document root <none> and password file <none>
I20260812 05:22:46.456248 20805 fs_manager.cc:359] Metadata directory not provided
I20260812 05:22:46.456297 20805 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 05:22:46.456415 20805 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 12 05:22:46 dist-test-slave-42z9 krb5kdc[19378](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512166, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.100.3@KRBTEST.COM for HTTP/127.31.100.20@KRBTEST.COM
I20260812 05:22:46.470496 20805 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786512017613366-32144-0/minicluster-data/ts-2/data/instance:
uuid: "1d4519ada40b4b53a1b03508ed47a1a0"
format_stamp: "Formatted at 2026-08-12 05:22:46 on dist-test-slave-42z9"
server_key: "6d3fc7a6677b24b9f384adb4a4ff52c9"
server_key_iv: "96d3d799c11c150e5633de0b0e05a479"
server_key_version: "kuduclusterkey@0"
I20260812 05:22:46.471021 20805 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786512017613366-32144-0/minicluster-data/ts-2/wal/instance:
uuid: "1d4519ada40b4b53a1b03508ed47a1a0"
format_stamp: "Formatted at 2026-08-12 05:22:46 on dist-test-slave-42z9"
server_key: "6d3fc7a6677b24b9f384adb4a4ff52c9"
server_key_iv: "96d3d799c11c150e5633de0b0e05a479"
server_key_version: "kuduclusterkey@0"
I20260812 05:22:46.474140 20805 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.001s
I20260812 05:22:46.493654 20834 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 05:22:46.493932 20805 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.001s sys 0.001s
I20260812 05:22:46.494074 20805 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786512017613366-32144-0/minicluster-data/ts-2/data,/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786512017613366-32144-0/minicluster-data/ts-2/wal
uuid: "1d4519ada40b4b53a1b03508ed47a1a0"
format_stamp: "Formatted at 2026-08-12 05:22:46 on dist-test-slave-42z9"
server_key: "6d3fc7a6677b24b9f384adb4a4ff52c9"
server_key_iv: "96d3d799c11c150e5633de0b0e05a479"
server_key_version: "kuduclusterkey@0"
I20260812 05:22:46.494230 20805 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786512017613366-32144-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786512017613366-32144-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786512017613366-32144-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 05:22:46.528795 20805 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 05:22:46.533528 20805 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 05:22:46.533699 20805 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 05:22:46.533960 20805 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 05:22:46.534379 20805 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 05:22:46.534447 20805 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 05:22:46.534482 20805 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 05:22:46.534554 20805 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 05:22:46.543138 20805 rpc_server.cc:307] RPC server started. Bound to: 127.31.100.3:36419
I20260812 05:22:46.543565 20805 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786512017613366-32144-0/minicluster-data/ts-2/data/info.pb
I20260812 05:22:46.545621 21017 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.100.3:36419 every 8 connection(s)
I20260812 05:22:46.549476 32144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0waFvT/build/release/bin/kudu as pid 20805
I20260812 05:22:46.549589 32144 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786512017613366-32144-0/minicluster-data/ts-2/wal/instance
I20260812 05:22:46.549789 32144 external_mini_cluster.cc:1428] Setting key 4715ed8c4d510e93d9ae879e8ed578e3
Aug 12 05:22:46 dist-test-slave-42z9 krb5kdc[19378](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512166, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.100.3@KRBTEST.COM for kudu/127.31.100.62@KRBTEST.COM
I20260812 05:22:46.555346 20555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:22:46.544324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.100.3:56109 (local address 127.31.100.62:42701)
0812 05:22:46.544369 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0812 05:22:46.544371 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 05:22:46.544641 (+ 270us) server_negotiation.cc:407] Connection header received
0812 05:22:46.545299 (+ 658us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:22:46.545300 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:22:46.545314 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:22:46.545394 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:22:46.546506 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:22:46.547259 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:22:46.548958 (+ 1699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:22:46.549249 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:22:46.549314 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:22:46.552071 (+ 2757us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 05:22:46.552072 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 05:22:46.552073 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 05:22:46.552081 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0812 05:22:46.553744 (+ 1663us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:22:46.554264 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:22:46.554265 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:22:46.554266 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:22:46.554328 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:22:46.554623 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:22:46.554623 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:22:46.554624 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:22:46.554749 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 05:22:46.554811 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0812 05:22:46.555169 (+ 358us) server_negotiation.cc:299] Negotiation successful
0812 05:22:46.555255 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260812 05:22:46.555675 21022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:22:46.544310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.62:42701 (local address 127.31.100.3:56109)
0812 05:22:46.544483 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0812 05:22:46.544531 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0812 05:22:46.545187 (+ 656us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:22:46.545489 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:22:46.545493 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:22:46.545968 (+ 475us) client_negotiation.cc:189] Negotiated authn=SASL
0812 05:22:46.546393 (+ 425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:22:46.546400 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:22:46.547366 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:22:46.547367 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:22:46.548864 (+ 1497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:22:46.548867 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:22:46.549432 (+ 565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:22:46.549434 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:22:46.549646 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:22:46.550266 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 05:22:46.550276 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0812 05:22:46.551963 (+ 1687us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 05:22:46.553842 (+ 1879us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 05:22:46.553850 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 05:22:46.553852 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 05:22:46.554177 (+ 325us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 05:22:46.554402 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 05:22:46.554402 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 05:22:46.554403 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 05:22:46.554551 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 05:22:46.554820 (+ 269us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 05:22:46.554827 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 05:22:46.555015 (+ 188us) client_negotiation.cc:769] Sending connection context
0812 05:22:46.555143 (+ 128us) client_negotiation.cc:240] Negotiation successful
0812 05:22:46.555322 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":85,"threads_started":1}
I20260812 05:22:46.556357 21018 heartbeater.cc:347] Connected to a master server at 127.31.100.62:42701
I20260812 05:22:46.556465 21018 heartbeater.cc:464] Registering TS with master...
I20260812 05:22:46.556648 21018 heartbeater.cc:511] Master 127.31.100.62:42701 requested a full tablet report, sending...
I20260812 05:22:46.557058 20192 ts_manager.cc:194] Registered new tserver with Master: 1d4519ada40b4b53a1b03508ed47a1a0 (127.31.100.3:36419)
I20260812 05:22:46.557549 20192 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.100.3@KRBTEST.COM'} at 127.31.100.3:56109
I20260812 05:22:46.561473 32144 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 05:22:46.562399 32144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0waFvT/build/release/bin/kudu with pid 20334
I20260812 05:22:46.573076 32144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0waFvT/build/release/bin/kudu with pid 20564
I20260812 05:22:46.585316 32144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0waFvT/build/release/bin/kudu with pid 20805
I20260812 05:22:46.620177 32144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0waFvT/build/release/bin/kudu with pid 20135
I20260812 05:22:46.628893 32144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0waFvT/build/release/bin/kudu
/tmp/dist-test-task0waFvT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786512017613366-32144-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786512017613366-32144-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786512017613366-32144-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786512017613366-32144-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.100.62:42701
--webserver_interface=127.31.100.62
--webserver_port=37371
--keytab_file=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.62.keytab
--principal=kudu/127.31.100.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.100.20:44553
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.100.62:42701
--ranger_config_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786512017613366-32144-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.31.100.20:56645/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.31.100.20:56645/kms with env {KRB5CCNAME=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786512017613366-32144-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786512017613366-32144-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786512017613366-32144-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 05:22:46.712275 21023 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 05:22:46.712484 21023 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 05:22:46.712504 21023 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 05:22:46.713663 21023 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 05:22:46.713703 21023 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 05:22:46.713716 21023 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 05:22:46.713727 21023 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 05:22:46.713738 21023 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 05:22:46.715288 21023 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.100.20:44553
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786512017613366-32144-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786512017613366-32144-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.31.100.20:56645/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.31.100.62:42701
--ranger_config_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786512017613366-32144-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.62.keytab
--principal=kudu/127.31.100.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.100.62:42701
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786512017613366-32144-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.100.62
--webserver_port=37371
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21023
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786512017613366-32144-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 04:20:34 UTC on ccycloud.quasar-phqdqp.root.comops.site
build id 4581871
I20260812 05:22:46.715512 21023 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 05:22:46.715715 21023 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 05:22:46.721787 21029 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 05:22:46.721906 21038 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 05:22:46.721800 21028 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 05:22:46.722103 21023 server_base.cc:1034] running on GCE node
I20260812 05:22:46.722347 21023 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 05:22:46.722667 21023 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 05:22:46.723836 21023 hybrid_clock.cc:648] HybridClock initialized: now 1786512166723809 us; error 40 us; skew 500 ppm
Aug 12 05:22:46 dist-test-slave-42z9 krb5kdc[19378](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512166, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.100.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 05:22:46.726814 21023 init.cc:377] Logged in from keytab as kudu/127.31.100.62@KRBTEST.COM (short username kudu)
I20260812 05:22:46.727326 21023 webserver.cc:460] Webserver started at http://127.31.100.62:37371/ using document root <none> and password file <none>
I20260812 05:22:46.727550 21023 fs_manager.cc:359] Metadata directory not provided
I20260812 05:22:46.727614 21023 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 12 05:22:46 dist-test-slave-42z9 krb5kdc[19378](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512166, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.100.62@KRBTEST.COM for HTTP/127.31.100.20@KRBTEST.COM
I20260812 05:22:46.766085 21023 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260812 05:22:46.767061 21054 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 05:22:46.767267 21023 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 05:22:46.767390 21023 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786512017613366-32144-0/minicluster-data/master-0/data,/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786512017613366-32144-0/minicluster-data/master-0/wal
uuid: "04405e5fcf124400abdb1c71ecfb4985"
format_stamp: "Formatted at 2026-08-12 05:22:44 on dist-test-slave-42z9"
server_key: "7d7f25bd8e55ccb891b4a6c6d6d47073"
server_key_iv: "8e9d96534a9f3b0c32935941fb130987"
server_key_version: "kuduclusterkey@0"
I20260812 05:22:46.767856 21023 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786512017613366-32144-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786512017613366-32144-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786512017613366-32144-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 05:22:46.783722 21023 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 05:22:46.785828 21023 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 05:22:46.785987 21023 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 05:22:46.791641 21023 rpc_server.cc:307] RPC server started. Bound to: 127.31.100.62:42701
I20260812 05:22:46.791669 21147 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.100.62:42701 every 8 connection(s)
I20260812 05:22:46.792156 21023 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786512017613366-32144-0/minicluster-data/master-0/data/info.pb
I20260812 05:22:46.793526 21148 sys_catalog.cc:263] Verifying existing consensus state
I20260812 05:22:46.794181 21148 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 04405e5fcf124400abdb1c71ecfb4985: Bootstrap starting.
I20260812 05:22:46.794678 32144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0waFvT/build/release/bin/kudu as pid 21023
I20260812 05:22:46.794991 32144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0waFvT/build/release/bin/kudu
/tmp/dist-test-task0waFvT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786512017613366-32144-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786512017613366-32144-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786512017613366-32144-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786512017613366-32144-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.31.100.20:56645/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.100.1:33025
--local_ip_for_outbound_sockets=127.31.100.1
--tserver_master_addrs=127.31.100.62:42701
--webserver_port=42803
--webserver_interface=127.31.100.1
--keytab_file=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.1.keytab
--principal=kudu/127.31.100.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.100.20:44553
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.31.100.20:56645/kms with env {KRB5CCNAME=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786512017613366-32144-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786512017613366-32144-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786512017613366-32144-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 05:22:46.802610 21148 log.cc:824] T 00000000000000000000000000000000 P 04405e5fcf124400abdb1c71ecfb4985: Log is configured to *not* fsync() on all Append() calls
I20260812 05:22:46.804185 21148 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 04405e5fcf124400abdb1c71ecfb4985: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 05:22:46.804454 21148 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 04405e5fcf124400abdb1c71ecfb4985: Bootstrap complete.
I20260812 05:22:46.806705 21148 raft_consensus.cc:348] T 00000000000000000000000000000000 P 04405e5fcf124400abdb1c71ecfb4985 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04405e5fcf124400abdb1c71ecfb4985" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 42701 } }
I20260812 05:22:46.806936 21148 raft_consensus.cc:729] T 00000000000000000000000000000000 P 04405e5fcf124400abdb1c71ecfb4985 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04405e5fcf124400abdb1c71ecfb4985, State: Initialized, Role: FOLLOWER
I20260812 05:22:46.807096 21148 consensus_queue.cc:260] T 00000000000000000000000000000000 P 04405e5fcf124400abdb1c71ecfb4985 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04405e5fcf124400abdb1c71ecfb4985" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 42701 } }
I20260812 05:22:46.807171 21148 raft_consensus.cc:388] T 00000000000000000000000000000000 P 04405e5fcf124400abdb1c71ecfb4985 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 05:22:46.807219 21148 raft_consensus.cc:482] T 00000000000000000000000000000000 P 04405e5fcf124400abdb1c71ecfb4985 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 05:22:46.807263 21148 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 04405e5fcf124400abdb1c71ecfb4985 [term 1 FOLLOWER]: Advancing to term 2
I20260812 05:22:46.808209 21148 raft_consensus.cc:504] T 00000000000000000000000000000000 P 04405e5fcf124400abdb1c71ecfb4985 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04405e5fcf124400abdb1c71ecfb4985" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 42701 } }
I20260812 05:22:46.808305 21148 leader_election.cc:302] T 00000000000000000000000000000000 P 04405e5fcf124400abdb1c71ecfb4985 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 04405e5fcf124400abdb1c71ecfb4985; no voters:
I20260812 05:22:46.808463 21148 leader_election.cc:288] T 00000000000000000000000000000000 P 04405e5fcf124400abdb1c71ecfb4985 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260812 05:22:46.808507 21152 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 04405e5fcf124400abdb1c71ecfb4985 [term 2 FOLLOWER]: Leader election won for term 2
I20260812 05:22:46.808619 21152 raft_consensus.cc:686] T 00000000000000000000000000000000 P 04405e5fcf124400abdb1c71ecfb4985 [term 2 LEADER]: Becoming Leader. State: Replica: 04405e5fcf124400abdb1c71ecfb4985, State: Running, Role: LEADER
I20260812 05:22:46.808722 21152 consensus_queue.cc:237] T 00000000000000000000000000000000 P 04405e5fcf124400abdb1c71ecfb4985 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04405e5fcf124400abdb1c71ecfb4985" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 42701 } }
I20260812 05:22:46.808758 21148 sys_catalog.cc:564] T 00000000000000000000000000000000 P 04405e5fcf124400abdb1c71ecfb4985 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 05:22:46.809017 21154 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04405e5fcf124400abdb1c71ecfb4985 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "04405e5fcf124400abdb1c71ecfb4985" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04405e5fcf124400abdb1c71ecfb4985" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 42701 } } }
I20260812 05:22:46.809147 21154 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04405e5fcf124400abdb1c71ecfb4985 [sys.catalog]: This master's current role is: LEADER
I20260812 05:22:46.809023 21152 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04405e5fcf124400abdb1c71ecfb4985 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 04405e5fcf124400abdb1c71ecfb4985. Latest consensus state: current_term: 2 leader_uuid: "04405e5fcf124400abdb1c71ecfb4985" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04405e5fcf124400abdb1c71ecfb4985" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 42701 } } }
I20260812 05:22:46.809250 21152 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04405e5fcf124400abdb1c71ecfb4985 [sys.catalog]: This master's current role is: LEADER
I20260812 05:22:46.809415 21163 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 05:22:46.809952 21163 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 05:22:46.810089 21163 catalog_manager.cc:1259] Loaded cluster ID: adfb6b5d097d44f292e52a24fc9c8dce
I20260812 05:22:46.810130 21163 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 05:22:46.810308 21148 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786512017613366-32144-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260812 05:22:46.814191 21163 catalog_manager.cc:1504] Loading token signing keys...
I20260812 05:22:46.814775 21163 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 04405e5fcf124400abdb1c71ecfb4985: Loaded TSK: 0
I20260812 05:22:46.815052 21163 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260812 05:22:46.910346 21150 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 05:22:46.910612 21150 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 05:22:46.910643 21150 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 05:22:46.912225 21150 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 05:22:46.912282 21150 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 05:22:46.912319 21150 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.100.1
I20260812 05:22:46.914407 21150 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.100.20:44553
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786512017613366-32144-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786512017613366-32144-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.31.100.20:56645/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.1.keytab
--principal=kudu/127.31.100.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.100.1:33025
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786512017613366-32144-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.100.1
--webserver_port=42803
--tserver_master_addrs=127.31.100.62:42701
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21150
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.100.1
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786512017613366-32144-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 04:20:34 UTC on ccycloud.quasar-phqdqp.root.comops.site
build id 4581871
I20260812 05:22:46.914637 21150 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 05:22:46.914880 21150 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 05:22:46.920145 21216 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 05:22:46.920259 21214 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 05:22:46.920434 21212 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 05:22:46.921114 21150 server_base.cc:1034] running on GCE node
I20260812 05:22:46.921315 21150 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 05:22:46.921590 21150 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 05:22:46.922751 21150 hybrid_clock.cc:648] HybridClock initialized: now 1786512166922738 us; error 26 us; skew 500 ppm
Aug 12 05:22:46 dist-test-slave-42z9 krb5kdc[19378](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512166, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.100.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 05:22:46.925678 21150 init.cc:377] Logged in from keytab as kudu/127.31.100.1@KRBTEST.COM (short username kudu)
I20260812 05:22:46.926141 21150 webserver.cc:460] Webserver started at http://127.31.100.1:42803/ using document root <none> and password file <none>
I20260812 05:22:46.926380 21150 fs_manager.cc:359] Metadata directory not provided
I20260812 05:22:46.926445 21150 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 12 05:22:46 dist-test-slave-42z9 krb5kdc[19378](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512166, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.100.1@KRBTEST.COM for HTTP/127.31.100.20@KRBTEST.COM
I20260812 05:22:46.950080 21150 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.001s
I20260812 05:22:46.954321 21228 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 05:22:46.954540 21150 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.000s
I20260812 05:22:46.954630 21150 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786512017613366-32144-0/minicluster-data/ts-0/data,/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786512017613366-32144-0/minicluster-data/ts-0/wal
uuid: "0502228f6e7340dea89eeaf2aa9f6e75"
format_stamp: "Formatted at 2026-08-12 05:22:45 on dist-test-slave-42z9"
server_key: "ba716f95e597fe7de1d7c4c125b2583e"
server_key_iv: "5a7f079e618cdb9afd0b009e84e57193"
server_key_version: "kuduclusterkey@0"
I20260812 05:22:46.954996 21150 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786512017613366-32144-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786512017613366-32144-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786512017613366-32144-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 05:22:46.979737 21150 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 05:22:46.981796 21150 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 05:22:46.981925 21150 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 05:22:46.982177 21150 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 05:22:46.982551 21150 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 05:22:46.982589 21150 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 05:22:46.982611 21150 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 05:22:46.982622 21150 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 05:22:46.989537 21150 rpc_server.cc:307] RPC server started. Bound to: 127.31.100.1:33025
I20260812 05:22:46.989717 21419 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.100.1:33025 every 8 connection(s)
I20260812 05:22:46.989969 21150 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786512017613366-32144-0/minicluster-data/ts-0/data/info.pb
I20260812 05:22:46.992375 32144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0waFvT/build/release/bin/kudu as pid 21150
I20260812 05:22:46.992686 32144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0waFvT/build/release/bin/kudu
/tmp/dist-test-task0waFvT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786512017613366-32144-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786512017613366-32144-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786512017613366-32144-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786512017613366-32144-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.31.100.20:56645/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.100.2:39845
--local_ip_for_outbound_sockets=127.31.100.2
--tserver_master_addrs=127.31.100.62:42701
--webserver_port=36851
--webserver_interface=127.31.100.2
--keytab_file=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.2.keytab
--principal=kudu/127.31.100.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.100.20:44553
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.31.100.20:56645/kms with env {KRB5CCNAME=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786512017613366-32144-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786512017613366-32144-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786512017613366-32144-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 12 05:22:46 dist-test-slave-42z9 krb5kdc[19378](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512166, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.100.1@KRBTEST.COM for kudu/127.31.100.62@KRBTEST.COM
I20260812 05:22:47.003490 21422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:22:46.990710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.62:42701 (local address 127.31.100.1:33581)
0812 05:22:46.990876 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0812 05:22:46.990918 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0812 05:22:46.991461 (+ 543us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:22:46.992120 (+ 659us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:22:46.992124 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:22:46.992343 (+ 219us) client_negotiation.cc:189] Negotiated authn=SASL
0812 05:22:46.992621 (+ 278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:22:46.992627 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:22:46.994332 (+ 1705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:22:46.994334 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:22:46.995587 (+ 1253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:22:46.995590 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:22:46.996192 (+ 602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:22:46.996193 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:22:46.996313 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:22:46.996803 (+ 490us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 05:22:46.996819 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0812 05:22:46.998325 (+ 1506us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 05:22:47.001667 (+ 3342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 05:22:47.001674 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 05:22:47.001676 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 05:22:47.001899 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 05:22:47.002195 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 05:22:47.002196 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 05:22:47.002197 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 05:22:47.002339 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 05:22:47.002834 (+ 495us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 05:22:47.002840 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 05:22:47.002986 (+ 146us) client_negotiation.cc:769] Sending connection context
0812 05:22:47.003077 (+ 91us) client_negotiation.cc:240] Negotiation successful
0812 05:22:47.003236 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":80,"threads_started":1}
I20260812 05:22:47.004076 21424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:22:46.990798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.100.1:33581 (local address 127.31.100.62:42701)
0812 05:22:46.991063 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0812 05:22:46.991073 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 05:22:46.991110 (+ 37us) server_negotiation.cc:407] Connection header received
0812 05:22:46.991762 (+ 652us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:22:46.991769 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:22:46.991825 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:22:46.991964 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:22:46.992765 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:22:46.994190 (+ 1425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:22:46.995661 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:22:46.996082 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:22:46.996179 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:22:46.999167 (+ 2988us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 05:22:46.999171 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 05:22:46.999171 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 05:22:46.999181 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0812 05:22:47.001564 (+ 2383us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:22:47.001998 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:22:47.002001 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:22:47.002001 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:22:47.002113 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:22:47.002440 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:22:47.002441 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:22:47.002441 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:22:47.002741 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 05:22:47.003668 (+ 927us) server_negotiation.cc:1035] Waiting for connection context
0812 05:22:47.003758 (+ 90us) server_negotiation.cc:299] Negotiation successful
0812 05:22:47.003883 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":210,"thread_start_us":82,"threads_started":1}
I20260812 05:22:47.004812 21420 heartbeater.cc:347] Connected to a master server at 127.31.100.62:42701
I20260812 05:22:47.004930 21420 heartbeater.cc:464] Registering TS with master...
I20260812 05:22:47.005134 21420 heartbeater.cc:511] Master 127.31.100.62:42701 requested a full tablet report, sending...
W20260812 05:22:47.009646 21420 heartbeater.cc:644] Failed to heartbeat to 127.31.100.62:42701 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260812 05:22:47.009732 21420 heartbeater.cc:464] Registering TS with master...
I20260812 05:22:47.009804 21420 heartbeater.cc:511] Master 127.31.100.62:42701 requested a full tablet report, sending...
I20260812 05:22:47.015563 21420 heartbeater.cc:464] Registering TS with master...
I20260812 05:22:47.015666 21420 heartbeater.cc:511] Master 127.31.100.62:42701 requested a full tablet report, sending...
W20260812 05:22:47.016011 21420 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260812 05:22:47.107782 21425 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 05:22:47.108011 21425 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 05:22:47.108048 21425 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 05:22:47.109700 21425 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 05:22:47.109750 21425 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 05:22:47.109786 21425 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.100.2
I20260812 05:22:47.111789 21425 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.100.20:44553
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786512017613366-32144-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786512017613366-32144-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.31.100.20:56645/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.2.keytab
--principal=kudu/127.31.100.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.100.2:39845
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786512017613366-32144-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.100.2
--webserver_port=36851
--tserver_master_addrs=127.31.100.62:42701
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21425
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.100.2
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786512017613366-32144-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 04:20:34 UTC on ccycloud.quasar-phqdqp.root.comops.site
build id 4581871
I20260812 05:22:47.112037 21425 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 05:22:47.112262 21425 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 05:22:47.118161 21432 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 05:22:47.118255 21436 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 05:22:47.118161 21431 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 05:22:47.122463 21425 server_base.cc:1034] running on GCE node
I20260812 05:22:47.122740 21425 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 05:22:47.122982 21425 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 05:22:47.124166 21425 hybrid_clock.cc:648] HybridClock initialized: now 1786512167124152 us; error 29 us; skew 500 ppm
Aug 12 05:22:47 dist-test-slave-42z9 krb5kdc[19378](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512167, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.100.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 05:22:47.126760 21425 init.cc:377] Logged in from keytab as kudu/127.31.100.2@KRBTEST.COM (short username kudu)
I20260812 05:22:47.127199 21425 webserver.cc:460] Webserver started at http://127.31.100.2:36851/ using document root <none> and password file <none>
I20260812 05:22:47.127408 21425 fs_manager.cc:359] Metadata directory not provided
I20260812 05:22:47.127467 21425 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 12 05:22:47 dist-test-slave-42z9 krb5kdc[19378](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512167, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.100.2@KRBTEST.COM for HTTP/127.31.100.20@KRBTEST.COM
I20260812 05:22:47.145828 21425 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.001s sys 0.000s
I20260812 05:22:47.146488 21453 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 05:22:47.146716 21425 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260812 05:22:47.146813 21425 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786512017613366-32144-0/minicluster-data/ts-1/data,/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786512017613366-32144-0/minicluster-data/ts-1/wal
uuid: "d80067ce5a344f35bf1996cf0f6e20c8"
format_stamp: "Formatted at 2026-08-12 05:22:46 on dist-test-slave-42z9"
server_key: "da654db2ff57b498a6b4c70e12d57584"
server_key_iv: "a04f9977bda44dd72b246644aafdd0f8"
server_key_version: "kuduclusterkey@0"
I20260812 05:22:47.147893 21425 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786512017613366-32144-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786512017613366-32144-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786512017613366-32144-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 05:22:47.164824 21425 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 05:22:47.166833 21425 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 05:22:47.166987 21425 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 05:22:47.167187 21425 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 05:22:47.167558 21425 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 05:22:47.167605 21425 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 05:22:47.167636 21425 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 05:22:47.167659 21425 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 05:22:47.174384 21425 rpc_server.cc:307] RPC server started. Bound to: 127.31.100.2:39845
I20260812 05:22:47.174453 21623 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.100.2:39845 every 8 connection(s)
I20260812 05:22:47.174827 21425 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786512017613366-32144-0/minicluster-data/ts-1/data/info.pb
Aug 12 05:22:47 dist-test-slave-42z9 krb5kdc[19378](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512167, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.100.2@KRBTEST.COM for kudu/127.31.100.62@KRBTEST.COM
I20260812 05:22:47.184330 32144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0waFvT/build/release/bin/kudu as pid 21425
I20260812 05:22:47.184644 32144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0waFvT/build/release/bin/kudu
/tmp/dist-test-task0waFvT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786512017613366-32144-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786512017613366-32144-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786512017613366-32144-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786512017613366-32144-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.31.100.20:56645/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.100.3:36419
--local_ip_for_outbound_sockets=127.31.100.3
--tserver_master_addrs=127.31.100.62:42701
--webserver_port=40811
--webserver_interface=127.31.100.3
--keytab_file=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.3.keytab
--principal=kudu/127.31.100.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.100.20:44553
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.31.100.20:56645/kms with env {KRB5CCNAME=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786512017613366-32144-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786512017613366-32144-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786512017613366-32144-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 05:22:47.185750 21424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:22:47.176048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.100.2:54099 (local address 127.31.100.62:42701)
0812 05:22:47.176094 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0812 05:22:47.176096 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 05:22:47.176106 (+ 10us) server_negotiation.cc:407] Connection header received
0812 05:22:47.176661 (+ 555us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:22:47.176662 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:22:47.176677 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:22:47.176729 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:22:47.177317 (+ 588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:22:47.177940 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:22:47.179401 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:22:47.179721 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:22:47.179797 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:22:47.182315 (+ 2518us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 05:22:47.182317 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 05:22:47.182317 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 05:22:47.182325 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0812 05:22:47.183910 (+ 1585us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:22:47.184373 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:22:47.184375 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:22:47.184375 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:22:47.184450 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:22:47.184986 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:22:47.184987 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:22:47.184988 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:22:47.185125 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 05:22:47.185167 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0812 05:22:47.185562 (+ 395us) server_negotiation.cc:299] Negotiation successful
0812 05:22:47.185653 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260812 05:22:47.186447 21627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:22:47.175715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.62:42701 (local address 127.31.100.2:54099)
0812 05:22:47.175885 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0812 05:22:47.175922 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0812 05:22:47.176547 (+ 625us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:22:47.176799 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:22:47.176803 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:22:47.177028 (+ 225us) client_negotiation.cc:189] Negotiated authn=SASL
0812 05:22:47.177223 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:22:47.177227 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:22:47.178033 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:22:47.178034 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:22:47.179290 (+ 1256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:22:47.179294 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:22:47.179845 (+ 551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:22:47.179847 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:22:47.179984 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:22:47.180554 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 05:22:47.180564 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0812 05:22:47.182216 (+ 1652us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 05:22:47.183998 (+ 1782us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 05:22:47.184004 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 05:22:47.184006 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 05:22:47.184281 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 05:22:47.184575 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 05:22:47.184575 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 05:22:47.184576 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 05:22:47.184901 (+ 325us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 05:22:47.185212 (+ 311us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 05:22:47.185218 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 05:22:47.185372 (+ 154us) client_negotiation.cc:769] Sending connection context
0812 05:22:47.186036 (+ 664us) client_negotiation.cc:240] Negotiation successful
0812 05:22:47.186203 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":100,"threads_started":1}
I20260812 05:22:47.187129 21625 heartbeater.cc:347] Connected to a master server at 127.31.100.62:42701
I20260812 05:22:47.187235 21625 heartbeater.cc:464] Registering TS with master...
I20260812 05:22:47.187426 21625 heartbeater.cc:511] Master 127.31.100.62:42701 requested a full tablet report, sending...
W20260812 05:22:47.188199 21625 heartbeater.cc:644] Failed to heartbeat to 127.31.100.62:42701 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260812 05:22:47.188283 21625 heartbeater.cc:464] Registering TS with master...
I20260812 05:22:47.188352 21625 heartbeater.cc:511] Master 127.31.100.62:42701 requested a full tablet report, sending...
I20260812 05:22:47.188730 21625 heartbeater.cc:464] Registering TS with master...
I20260812 05:22:47.188851 21625 heartbeater.cc:511] Master 127.31.100.62:42701 requested a full tablet report, sending...
W20260812 05:22:47.189307 21625 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260812 05:22:47.290115 21629 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 05:22:47.290375 21629 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 05:22:47.290405 21629 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 05:22:47.291944 21629 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 05:22:47.292002 21629 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 05:22:47.292037 21629 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.100.3
I20260812 05:22:47.294056 21629 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.100.20:44553
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786512017613366-32144-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786512017613366-32144-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.31.100.20:56645/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.3.keytab
--principal=kudu/127.31.100.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.100.3:36419
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786512017613366-32144-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.100.3
--webserver_port=40811
--tserver_master_addrs=127.31.100.62:42701
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21629
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.100.3
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786512017613366-32144-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 04:20:34 UTC on ccycloud.quasar-phqdqp.root.comops.site
build id 4581871
I20260812 05:22:47.294298 21629 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 05:22:47.294579 21629 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 05:22:47.300189 21643 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 05:22:47.301609 21639 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 05:22:47.303325 21638 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 05:22:47.303503 21629 server_base.cc:1034] running on GCE node
I20260812 05:22:47.303737 21629 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 05:22:47.304033 21629 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 05:22:47.305208 21629 hybrid_clock.cc:648] HybridClock initialized: now 1786512167305191 us; error 29 us; skew 500 ppm
Aug 12 05:22:47 dist-test-slave-42z9 krb5kdc[19378](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512167, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.100.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 05:22:47.308328 21629 init.cc:377] Logged in from keytab as kudu/127.31.100.3@KRBTEST.COM (short username kudu)
I20260812 05:22:47.308911 21629 webserver.cc:460] Webserver started at http://127.31.100.3:40811/ using document root <none> and password file <none>
I20260812 05:22:47.309266 21629 fs_manager.cc:359] Metadata directory not provided
I20260812 05:22:47.309405 21629 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 12 05:22:47 dist-test-slave-42z9 krb5kdc[19378](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512167, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.100.3@KRBTEST.COM for HTTP/127.31.100.20@KRBTEST.COM
I20260812 05:22:47.331621 21629 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260812 05:22:47.332342 21656 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 05:22:47.332525 21629 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 05:22:47.332633 21629 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786512017613366-32144-0/minicluster-data/ts-2/data,/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786512017613366-32144-0/minicluster-data/ts-2/wal
uuid: "1d4519ada40b4b53a1b03508ed47a1a0"
format_stamp: "Formatted at 2026-08-12 05:22:46 on dist-test-slave-42z9"
server_key: "6d3fc7a6677b24b9f384adb4a4ff52c9"
server_key_iv: "96d3d799c11c150e5633de0b0e05a479"
server_key_version: "kuduclusterkey@0"
I20260812 05:22:47.333106 21629 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786512017613366-32144-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786512017613366-32144-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786512017613366-32144-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 05:22:47.349121 21629 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 05:22:47.351037 21629 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 05:22:47.351181 21629 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 05:22:47.351440 21629 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 05:22:47.351811 21629 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 05:22:47.351859 21629 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 05:22:47.351881 21629 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 05:22:47.351893 21629 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 05:22:47.358727 21629 rpc_server.cc:307] RPC server started. Bound to: 127.31.100.3:36419
I20260812 05:22:47.358930 21867 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.100.3:36419 every 8 connection(s)
I20260812 05:22:47.359138 21629 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786512017613366-32144-0/minicluster-data/ts-2/data/info.pb
Aug 12 05:22:47 dist-test-slave-42z9 krb5kdc[19378](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512167, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.100.3@KRBTEST.COM for kudu/127.31.100.62@KRBTEST.COM
I20260812 05:22:47.366895 32144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0waFvT/build/release/bin/kudu as pid 21629
I20260812 05:22:47.368790 21424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:22:47.359853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.100.3:49975 (local address 127.31.100.62:42701)
0812 05:22:47.359895 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0812 05:22:47.359897 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 05:22:47.360124 (+ 227us) server_negotiation.cc:407] Connection header received
0812 05:22:47.360626 (+ 502us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:22:47.360628 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:22:47.360641 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:22:47.360705 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:22:47.361173 (+ 468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:22:47.361970 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:22:47.363288 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:22:47.363601 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:22:47.363665 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:22:47.365759 (+ 2094us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 05:22:47.365760 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 05:22:47.365761 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 05:22:47.365768 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 05:22:47.367331 (+ 1563us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:22:47.367790 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:22:47.367791 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:22:47.367791 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:22:47.367856 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:22:47.368175 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:22:47.368176 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:22:47.368176 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:22:47.368290 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 05:22:47.368323 (+ 33us) server_negotiation.cc:1035] Waiting for connection context
0812 05:22:47.368612 (+ 289us) server_negotiation.cc:299] Negotiation successful
0812 05:22:47.368700 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260812 05:22:47.369161 21878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:22:47.359855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.62:42701 (local address 127.31.100.3:49975)
0812 05:22:47.359996 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0812 05:22:47.360030 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0812 05:22:47.360534 (+ 504us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:22:47.360758 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:22:47.360761 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:22:47.360922 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0812 05:22:47.361107 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:22:47.361111 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:22:47.362063 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:22:47.362064 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:22:47.363197 (+ 1133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:22:47.363201 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:22:47.363697 (+ 496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:22:47.363698 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:22:47.363806 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:22:47.364253 (+ 447us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 05:22:47.364261 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0812 05:22:47.365666 (+ 1405us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 05:22:47.367407 (+ 1741us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 05:22:47.367414 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 05:22:47.367416 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 05:22:47.367700 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 05:22:47.367936 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 05:22:47.367938 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 05:22:47.367938 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 05:22:47.368097 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 05:22:47.368359 (+ 262us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 05:22:47.368364 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 05:22:47.368480 (+ 116us) client_negotiation.cc:769] Sending connection context
0812 05:22:47.368565 (+ 85us) client_negotiation.cc:240] Negotiation successful
0812 05:22:47.368701 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":77,"threads_started":1}
I20260812 05:22:47.369877 21873 heartbeater.cc:347] Connected to a master server at 127.31.100.62:42701
I20260812 05:22:47.369984 21873 heartbeater.cc:464] Registering TS with master...
I20260812 05:22:47.370170 21873 heartbeater.cc:511] Master 127.31.100.62:42701 requested a full tablet report, sending...
W20260812 05:22:47.370855 21873 heartbeater.cc:644] Failed to heartbeat to 127.31.100.62:42701 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260812 05:22:47.370936 21873 heartbeater.cc:464] Registering TS with master...
I20260812 05:22:47.371025 21873 heartbeater.cc:511] Master 127.31.100.62:42701 requested a full tablet report, sending...
I20260812 05:22:47.373418 21873 heartbeater.cc:464] Registering TS with master...
I20260812 05:22:47.373612 21873 heartbeater.cc:511] Master 127.31.100.62:42701 requested a full tablet report, sending...
W20260812 05:22:47.373961 21873 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260812 05:22:47.380041 21880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:22:47.367288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57922 (local address 127.31.100.62:42701)
0812 05:22:47.368039 (+ 751us) server_negotiation.cc:206] Beginning negotiation
0812 05:22:47.368040 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0812 05:22:47.368047 (+ 7us) server_negotiation.cc:407] Connection header received
0812 05:22:47.368066 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:22:47.368066 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:22:47.368077 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:22:47.368146 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:22:47.368925 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:22:47.374111 (+ 5186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:22:47.375192 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:22:47.375537 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:22:47.375609 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:22:47.377061 (+ 1452us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 05:22:47.377062 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 05:22:47.377063 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 05:22:47.377069 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0812 05:22:47.378846 (+ 1777us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:22:47.379250 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:22:47.379251 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:22:47.379251 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:22:47.379306 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:22:47.379560 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:22:47.379561 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:22:47.379561 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:22:47.379658 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 05:22:47.379787 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0812 05:22:47.379875 (+ 88us) server_negotiation.cc:299] Negotiation successful
0812 05:22:47.379954 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":721,"thread_start_us":56,"threads_started":1}
Aug 12 05:22:47 dist-test-slave-42z9 krb5kdc[19378](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786512167, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.100.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 05:22:48.016199 21420 heartbeater.cc:464] Registering TS with master...
I20260812 05:22:48.016353 21420 heartbeater.cc:511] Master 127.31.100.62:42701 requested a full tablet report, sending...
I20260812 05:22:48.016726 21081 ts_manager.cc:194] Registered new tserver with Master: 0502228f6e7340dea89eeaf2aa9f6e75 (127.31.100.1:33025)
I20260812 05:22:48.017764 21081 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.100.1@KRBTEST.COM'} at 127.31.100.1:33581
I20260812 05:22:48.189514 21625 heartbeater.cc:464] Registering TS with master...
I20260812 05:22:48.189671 21625 heartbeater.cc:511] Master 127.31.100.62:42701 requested a full tablet report, sending...
I20260812 05:22:48.190092 21081 ts_manager.cc:194] Registered new tserver with Master: d80067ce5a344f35bf1996cf0f6e20c8 (127.31.100.2:39845)
I20260812 05:22:48.190681 21081 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.100.2@KRBTEST.COM'} at 127.31.100.2:54099
Aug 12 05:22:48 dist-test-slave-42z9 krb5kdc[19378](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512167, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.100.62@KRBTEST.COM for HTTP/127.31.100.20@KRBTEST.COM
I20260812 05:22:48.374186 21873 heartbeater.cc:464] Registering TS with master...
I20260812 05:22:48.374348 21873 heartbeater.cc:511] Master 127.31.100.62:42701 requested a full tablet report, sending...
I20260812 05:22:48.374735 21081 ts_manager.cc:194] Registered new tserver with Master: 1d4519ada40b4b53a1b03508ed47a1a0 (127.31.100.3:36419)
I20260812 05:22:48.375267 21081 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.100.3@KRBTEST.COM'} at 127.31.100.3:49975
I20260812 05:22:48.383824 32144 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 05:22:48.397652 21907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:22:48.386112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57938 (local address 127.31.100.62:42701)
0812 05:22:48.389777 (+ 3665us) server_negotiation.cc:206] Beginning negotiation
0812 05:22:48.389779 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 05:22:48.389786 (+ 7us) server_negotiation.cc:407] Connection header received
0812 05:22:48.389809 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:22:48.389810 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:22:48.389821 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:22:48.390658 (+ 837us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:22:48.390702 (+ 44us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:22:48.391499 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:22:48.392544 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:22:48.392873 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:22:48.393297 (+ 424us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:22:48.394485 (+ 1188us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 05:22:48.394486 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 05:22:48.394487 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 05:22:48.394494 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 05:22:48.396221 (+ 1727us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:22:48.396606 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:22:48.396607 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:22:48.396608 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:22:48.396663 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:22:48.396840 (+ 177us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:22:48.396841 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:22:48.396841 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:22:48.396976 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 05:22:48.397397 (+ 421us) server_negotiation.cc:1035] Waiting for connection context
0812 05:22:48.397463 (+ 66us) server_negotiation.cc:299] Negotiation successful
0812 05:22:48.397545 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3644,"thread_start_us":93,"threads_started":1}
I20260812 05:22:48.400104 21081 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57938:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 05:22:48.402346 21081 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 05:22:48.412189 21920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:22:48.405965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.2:39845 (local address 127.0.0.1:50590)
0812 05:22:48.407407 (+ 1442us) negotiation.cc:107] Waiting for socket to connect
0812 05:22:48.407417 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0812 05:22:48.407469 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:22:48.407825 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:22:48.407828 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:22:48.407830 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 05:22:48.407905 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:22:48.407906 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:22:48.409274 (+ 1368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:22:48.409275 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:22:48.410681 (+ 1406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:22:48.410685 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:22:48.411886 (+ 1201us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:22:48.411887 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:22:48.411973 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:22:48.411986 (+ 13us) client_negotiation.cc:769] Sending connection context
0812 05:22:48.412053 (+ 67us) client_negotiation.cc:240] Negotiation successful
0812 05:22:48.412102 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1403,"thread_start_us":119,"threads_started":1}
I20260812 05:22:48.412631 21924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:22:48.406564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50590 (local address 127.31.100.2:39845)
0812 05:22:48.407243 (+ 679us) server_negotiation.cc:206] Beginning negotiation
0812 05:22:48.407248 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 05:22:48.407524 (+ 276us) server_negotiation.cc:407] Connection header received
0812 05:22:48.407561 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:22:48.407564 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:22:48.407580 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:22:48.407643 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 05:22:48.407992 (+ 349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:22:48.409159 (+ 1167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:22:48.410775 (+ 1616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:22:48.411661 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:22:48.411778 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:22:48.411839 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0812 05:22:48.412478 (+ 639us) server_negotiation.cc:299] Negotiation successful
0812 05:22:48.412523 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":646,"thread_start_us":53,"threads_started":1}
I20260812 05:22:48.413666 21522 tablet_service.cc:1467] Processing CreateTablet for tablet f81ecb65858549558f4ccda333a4a061 (DEFAULT_TABLE table=test-table [id=b883decdc1a146f6ac4f2d0a862c43c6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 05:22:48.414060 21522 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f81ecb65858549558f4ccda333a4a061. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 05:22:48.418972 21926 tablet_bootstrap.cc:492] T f81ecb65858549558f4ccda333a4a061 P d80067ce5a344f35bf1996cf0f6e20c8: Bootstrap starting.
I20260812 05:22:48.420264 21926 tablet_bootstrap.cc:654] T f81ecb65858549558f4ccda333a4a061 P d80067ce5a344f35bf1996cf0f6e20c8: Neither blocks nor log segments found. Creating new log.
I20260812 05:22:48.421110 21926 log.cc:824] T f81ecb65858549558f4ccda333a4a061 P d80067ce5a344f35bf1996cf0f6e20c8: Log is configured to *not* fsync() on all Append() calls
I20260812 05:22:48.421209 21922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:22:48.405661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.3:36419 (local address 127.0.0.1:47880)
0812 05:22:48.406989 (+ 1328us) negotiation.cc:107] Waiting for socket to connect
0812 05:22:48.407030 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0812 05:22:48.407096 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:22:48.413875 (+ 6779us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:22:48.413876 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:22:48.413878 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 05:22:48.413985 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:22:48.413987 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:22:48.418390 (+ 4403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:22:48.418391 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:22:48.419586 (+ 1195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:22:48.419589 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:22:48.420669 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:22:48.420670 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:22:48.420777 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:22:48.420783 (+ 6us) client_negotiation.cc:769] Sending connection context
0812 05:22:48.420815 (+ 32us) client_negotiation.cc:240] Negotiation successful
0812 05:22:48.420852 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1282,"thread_start_us":133,"threads_started":1}
I20260812 05:22:48.421484 21917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:22:48.406170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.1:33025 (local address 127.0.0.1:45964)
0812 05:22:48.417508 (+ 11338us) negotiation.cc:107] Waiting for socket to connect
0812 05:22:48.417520 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0812 05:22:48.417615 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:22:48.417917 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:22:48.417918 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:22:48.417920 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 05:22:48.417965 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:22:48.417966 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:22:48.419153 (+ 1187us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:22:48.419154 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:22:48.420221 (+ 1067us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:22:48.420225 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:22:48.421235 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:22:48.421236 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:22:48.421294 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:22:48.421298 (+ 4us) client_negotiation.cc:769] Sending connection context
0812 05:22:48.421341 (+ 43us) client_negotiation.cc:240] Negotiation successful
0812 05:22:48.421392 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11319,"thread_start_us":181,"threads_started":1}
I20260812 05:22:48.421815 21918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:22:48.405768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47880 (local address 127.31.100.3:36419)
0812 05:22:48.413695 (+ 7927us) server_negotiation.cc:206] Beginning negotiation
0812 05:22:48.413700 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 05:22:48.413712 (+ 12us) server_negotiation.cc:407] Connection header received
0812 05:22:48.413746 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:22:48.413748 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:22:48.413764 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:22:48.417064 (+ 3300us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 05:22:48.417193 (+ 129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:22:48.418275 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:22:48.419694 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:22:48.420571 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:22:48.421577 (+ 1006us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:22:48.421642 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0812 05:22:48.421670 (+ 28us) server_negotiation.cc:299] Negotiation successful
0812 05:22:48.421712 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7889,"thread_start_us":91,"threads_started":1}
I20260812 05:22:48.421914 21925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:22:48.406694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45964 (local address 127.31.100.1:33025)
0812 05:22:48.407307 (+ 613us) server_negotiation.cc:206] Beginning negotiation
0812 05:22:48.407311 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 05:22:48.417586 (+ 10275us) server_negotiation.cc:407] Connection header received
0812 05:22:48.417702 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:22:48.417705 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:22:48.417722 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:22:48.417787 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 05:22:48.418054 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:22:48.419060 (+ 1006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:22:48.420328 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:22:48.421068 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:22:48.421116 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:22:48.421172 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0812 05:22:48.421796 (+ 624us) server_negotiation.cc:299] Negotiation successful
0812 05:22:48.421838 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":585,"thread_start_us":114,"threads_started":1}
I20260812 05:22:48.422334 21926 tablet_bootstrap.cc:492] T f81ecb65858549558f4ccda333a4a061 P d80067ce5a344f35bf1996cf0f6e20c8: No bootstrap required, opened a new log
I20260812 05:22:48.422407 21926 ts_tablet_manager.cc:1397] T f81ecb65858549558f4ccda333a4a061 P d80067ce5a344f35bf1996cf0f6e20c8: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.001s
I20260812 05:22:48.422676 21714 tablet_service.cc:1467] Processing CreateTablet for tablet f81ecb65858549558f4ccda333a4a061 (DEFAULT_TABLE table=test-table [id=b883decdc1a146f6ac4f2d0a862c43c6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 05:22:48.422667 21274 tablet_service.cc:1467] Processing CreateTablet for tablet f81ecb65858549558f4ccda333a4a061 (DEFAULT_TABLE table=test-table [id=b883decdc1a146f6ac4f2d0a862c43c6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 05:22:48.422999 21714 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f81ecb65858549558f4ccda333a4a061. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 05:22:48.423053 21274 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f81ecb65858549558f4ccda333a4a061. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 05:22:48.426527 21926 raft_consensus.cc:348] T f81ecb65858549558f4ccda333a4a061 P d80067ce5a344f35bf1996cf0f6e20c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d80067ce5a344f35bf1996cf0f6e20c8" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 39845 } } peers { permanent_uuid: "1d4519ada40b4b53a1b03508ed47a1a0" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 36419 } } peers { permanent_uuid: "0502228f6e7340dea89eeaf2aa9f6e75" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 33025 } }
I20260812 05:22:48.426766 21926 raft_consensus.cc:374] T f81ecb65858549558f4ccda333a4a061 P d80067ce5a344f35bf1996cf0f6e20c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 05:22:48.426846 21926 raft_consensus.cc:729] T f81ecb65858549558f4ccda333a4a061 P d80067ce5a344f35bf1996cf0f6e20c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d80067ce5a344f35bf1996cf0f6e20c8, State: Initialized, Role: FOLLOWER
I20260812 05:22:48.427048 21926 consensus_queue.cc:260] T f81ecb65858549558f4ccda333a4a061 P d80067ce5a344f35bf1996cf0f6e20c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d80067ce5a344f35bf1996cf0f6e20c8" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 39845 } } peers { permanent_uuid: "1d4519ada40b4b53a1b03508ed47a1a0" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 36419 } } peers { permanent_uuid: "0502228f6e7340dea89eeaf2aa9f6e75" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 33025 } }
I20260812 05:22:48.427512 21926 ts_tablet_manager.cc:1428] T f81ecb65858549558f4ccda333a4a061 P d80067ce5a344f35bf1996cf0f6e20c8: Time spent starting tablet: real 0.005s user 0.002s sys 0.003s
I20260812 05:22:48.427678 21625 heartbeater.cc:503] Master 127.31.100.62:42701 was elected leader, sending a full tablet report...
I20260812 05:22:48.432745 21934 tablet_bootstrap.cc:492] T f81ecb65858549558f4ccda333a4a061 P 0502228f6e7340dea89eeaf2aa9f6e75: Bootstrap starting.
I20260812 05:22:48.433534 21934 tablet_bootstrap.cc:654] T f81ecb65858549558f4ccda333a4a061 P 0502228f6e7340dea89eeaf2aa9f6e75: Neither blocks nor log segments found. Creating new log.
I20260812 05:22:48.433923 21934 log.cc:824] T f81ecb65858549558f4ccda333a4a061 P 0502228f6e7340dea89eeaf2aa9f6e75: Log is configured to *not* fsync() on all Append() calls
I20260812 05:22:48.434419 21930 tablet_bootstrap.cc:492] T f81ecb65858549558f4ccda333a4a061 P 1d4519ada40b4b53a1b03508ed47a1a0: Bootstrap starting.
I20260812 05:22:48.435216 21930 tablet_bootstrap.cc:654] T f81ecb65858549558f4ccda333a4a061 P 1d4519ada40b4b53a1b03508ed47a1a0: Neither blocks nor log segments found. Creating new log.
I20260812 05:22:48.435578 21930 log.cc:824] T f81ecb65858549558f4ccda333a4a061 P 1d4519ada40b4b53a1b03508ed47a1a0: Log is configured to *not* fsync() on all Append() calls
I20260812 05:22:48.436522 21934 tablet_bootstrap.cc:492] T f81ecb65858549558f4ccda333a4a061 P 0502228f6e7340dea89eeaf2aa9f6e75: No bootstrap required, opened a new log
I20260812 05:22:48.436635 21934 ts_tablet_manager.cc:1397] T f81ecb65858549558f4ccda333a4a061 P 0502228f6e7340dea89eeaf2aa9f6e75: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20260812 05:22:48.440502 21934 raft_consensus.cc:348] T f81ecb65858549558f4ccda333a4a061 P 0502228f6e7340dea89eeaf2aa9f6e75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d80067ce5a344f35bf1996cf0f6e20c8" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 39845 } } peers { permanent_uuid: "1d4519ada40b4b53a1b03508ed47a1a0" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 36419 } } peers { permanent_uuid: "0502228f6e7340dea89eeaf2aa9f6e75" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 33025 } }
I20260812 05:22:48.440729 21934 raft_consensus.cc:374] T f81ecb65858549558f4ccda333a4a061 P 0502228f6e7340dea89eeaf2aa9f6e75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 05:22:48.440807 21934 raft_consensus.cc:729] T f81ecb65858549558f4ccda333a4a061 P 0502228f6e7340dea89eeaf2aa9f6e75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0502228f6e7340dea89eeaf2aa9f6e75, State: Initialized, Role: FOLLOWER
I20260812 05:22:48.441004 21934 consensus_queue.cc:260] T f81ecb65858549558f4ccda333a4a061 P 0502228f6e7340dea89eeaf2aa9f6e75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d80067ce5a344f35bf1996cf0f6e20c8" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 39845 } } peers { permanent_uuid: "1d4519ada40b4b53a1b03508ed47a1a0" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 36419 } } peers { permanent_uuid: "0502228f6e7340dea89eeaf2aa9f6e75" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 33025 } }
I20260812 05:22:48.441324 21934 ts_tablet_manager.cc:1428] T f81ecb65858549558f4ccda333a4a061 P 0502228f6e7340dea89eeaf2aa9f6e75: Time spent starting tablet: real 0.005s user 0.000s sys 0.003s
I20260812 05:22:48.441493 21420 heartbeater.cc:503] Master 127.31.100.62:42701 was elected leader, sending a full tablet report...
I20260812 05:22:48.443188 21930 tablet_bootstrap.cc:492] T f81ecb65858549558f4ccda333a4a061 P 1d4519ada40b4b53a1b03508ed47a1a0: No bootstrap required, opened a new log
I20260812 05:22:48.443255 21930 ts_tablet_manager.cc:1397] T f81ecb65858549558f4ccda333a4a061 P 1d4519ada40b4b53a1b03508ed47a1a0: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20260812 05:22:48.445788 21930 raft_consensus.cc:348] T f81ecb65858549558f4ccda333a4a061 P 1d4519ada40b4b53a1b03508ed47a1a0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d80067ce5a344f35bf1996cf0f6e20c8" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 39845 } } peers { permanent_uuid: "1d4519ada40b4b53a1b03508ed47a1a0" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 36419 } } peers { permanent_uuid: "0502228f6e7340dea89eeaf2aa9f6e75" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 33025 } }
I20260812 05:22:48.445950 21930 raft_consensus.cc:374] T f81ecb65858549558f4ccda333a4a061 P 1d4519ada40b4b53a1b03508ed47a1a0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 05:22:48.445981 21930 raft_consensus.cc:729] T f81ecb65858549558f4ccda333a4a061 P 1d4519ada40b4b53a1b03508ed47a1a0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d4519ada40b4b53a1b03508ed47a1a0, State: Initialized, Role: FOLLOWER
I20260812 05:22:48.446228 21930 consensus_queue.cc:260] T f81ecb65858549558f4ccda333a4a061 P 1d4519ada40b4b53a1b03508ed47a1a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d80067ce5a344f35bf1996cf0f6e20c8" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 39845 } } peers { permanent_uuid: "1d4519ada40b4b53a1b03508ed47a1a0" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 36419 } } peers { permanent_uuid: "0502228f6e7340dea89eeaf2aa9f6e75" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 33025 } }
I20260812 05:22:48.446614 21930 ts_tablet_manager.cc:1428] T f81ecb65858549558f4ccda333a4a061 P 1d4519ada40b4b53a1b03508ed47a1a0: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260812 05:22:48.446774 21873 heartbeater.cc:503] Master 127.31.100.62:42701 was elected leader, sending a full tablet report...
W20260812 05:22:48.491449 21421 tablet.cc:2369] T f81ecb65858549558f4ccda333a4a061 P 0502228f6e7340dea89eeaf2aa9f6e75: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 05:22:48.586648 21941 raft_consensus.cc:482] T f81ecb65858549558f4ccda333a4a061 P 1d4519ada40b4b53a1b03508ed47a1a0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 05:22:48.586850 21941 raft_consensus.cc:504] T f81ecb65858549558f4ccda333a4a061 P 1d4519ada40b4b53a1b03508ed47a1a0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d80067ce5a344f35bf1996cf0f6e20c8" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 39845 } } peers { permanent_uuid: "1d4519ada40b4b53a1b03508ed47a1a0" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 36419 } } peers { permanent_uuid: "0502228f6e7340dea89eeaf2aa9f6e75" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 33025 } }
I20260812 05:22:48.587393 21941 leader_election.cc:288] T f81ecb65858549558f4ccda333a4a061 P 1d4519ada40b4b53a1b03508ed47a1a0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d80067ce5a344f35bf1996cf0f6e20c8 (127.31.100.2:39845), 0502228f6e7340dea89eeaf2aa9f6e75 (127.31.100.1:33025)
I20260812 05:22:48.598070 21924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:22:48.588357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.100.3:49965 (local address 127.31.100.2:39845)
0812 05:22:48.588400 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0812 05:22:48.588402 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 05:22:48.591234 (+ 2832us) server_negotiation.cc:407] Connection header received
0812 05:22:48.591339 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:22:48.591341 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:22:48.591351 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:22:48.591379 (+ 28us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 05:22:48.591676 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:22:48.592431 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:22:48.596953 (+ 4522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:22:48.597557 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:22:48.597606 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:22:48.597615 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0812 05:22:48.597947 (+ 332us) server_negotiation.cc:299] Negotiation successful
0812 05:22:48.597995 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260812 05:22:48.598628 21945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:22:48.587619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.2:39845 (local address 127.31.100.3:49965)
0812 05:22:48.591153 (+ 3534us) negotiation.cc:107] Waiting for socket to connect
0812 05:22:48.591167 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0812 05:22:48.591267 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:22:48.591446 (+ 179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:22:48.591447 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:22:48.591449 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 05:22:48.591593 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:22:48.591595 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:22:48.592532 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:22:48.592533 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:22:48.594070 (+ 1537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:22:48.594074 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:22:48.597731 (+ 3657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:22:48.597732 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:22:48.597848 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:22:48.597852 (+ 4us) client_negotiation.cc:769] Sending connection context
0812 05:22:48.598374 (+ 522us) client_negotiation.cc:240] Negotiation successful
0812 05:22:48.598421 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3496,"thread_start_us":150,"threads_started":1}
I20260812 05:22:48.600194 21925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:22:48.588196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.100.3:34191 (local address 127.31.100.1:33025)
0812 05:22:48.588245 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0812 05:22:48.588247 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 05:22:48.594244 (+ 5997us) server_negotiation.cc:407] Connection header received
0812 05:22:48.594338 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:22:48.594340 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:22:48.594349 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:22:48.594376 (+ 27us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 05:22:48.594649 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:22:48.595361 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:22:48.598871 (+ 3510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:22:48.599515 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:22:48.599563 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:22:48.599570 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0812 05:22:48.600084 (+ 514us) server_negotiation.cc:299] Negotiation successful
0812 05:22:48.600130 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260812 05:22:48.600306 21532 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f81ecb65858549558f4ccda333a4a061" candidate_uuid: "1d4519ada40b4b53a1b03508ed47a1a0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d80067ce5a344f35bf1996cf0f6e20c8" is_pre_election: true
I20260812 05:22:48.600495 21532 raft_consensus.cc:2454] T f81ecb65858549558f4ccda333a4a061 P d80067ce5a344f35bf1996cf0f6e20c8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d4519ada40b4b53a1b03508ed47a1a0 in term 0.
I20260812 05:22:48.600790 21659 leader_election.cc:302] T f81ecb65858549558f4ccda333a4a061 P 1d4519ada40b4b53a1b03508ed47a1a0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d4519ada40b4b53a1b03508ed47a1a0, d80067ce5a344f35bf1996cf0f6e20c8; no voters:
I20260812 05:22:48.600926 21941 raft_consensus.cc:2781] T f81ecb65858549558f4ccda333a4a061 P 1d4519ada40b4b53a1b03508ed47a1a0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 05:22:48.600977 21941 raft_consensus.cc:482] T f81ecb65858549558f4ccda333a4a061 P 1d4519ada40b4b53a1b03508ed47a1a0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 05:22:48.600997 21941 raft_consensus.cc:3037] T f81ecb65858549558f4ccda333a4a061 P 1d4519ada40b4b53a1b03508ed47a1a0 [term 0 FOLLOWER]: Advancing to term 1
I20260812 05:22:48.602087 21941 raft_consensus.cc:504] T f81ecb65858549558f4ccda333a4a061 P 1d4519ada40b4b53a1b03508ed47a1a0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d80067ce5a344f35bf1996cf0f6e20c8" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 39845 } } peers { permanent_uuid: "1d4519ada40b4b53a1b03508ed47a1a0" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 36419 } } peers { permanent_uuid: "0502228f6e7340dea89eeaf2aa9f6e75" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 33025 } }
I20260812 05:22:48.602210 21941 leader_election.cc:288] T f81ecb65858549558f4ccda333a4a061 P 1d4519ada40b4b53a1b03508ed47a1a0 [CANDIDATE]: Term 1 election: Requested vote from peers d80067ce5a344f35bf1996cf0f6e20c8 (127.31.100.2:39845), 0502228f6e7340dea89eeaf2aa9f6e75 (127.31.100.1:33025)
I20260812 05:22:48.602499 21532 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f81ecb65858549558f4ccda333a4a061" candidate_uuid: "1d4519ada40b4b53a1b03508ed47a1a0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d80067ce5a344f35bf1996cf0f6e20c8"
I20260812 05:22:48.602573 21532 raft_consensus.cc:3037] T f81ecb65858549558f4ccda333a4a061 P d80067ce5a344f35bf1996cf0f6e20c8 [term 0 FOLLOWER]: Advancing to term 1
I20260812 05:22:48.603469 21532 raft_consensus.cc:2454] T f81ecb65858549558f4ccda333a4a061 P d80067ce5a344f35bf1996cf0f6e20c8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d4519ada40b4b53a1b03508ed47a1a0 in term 1.
I20260812 05:22:48.603693 21659 leader_election.cc:302] T f81ecb65858549558f4ccda333a4a061 P 1d4519ada40b4b53a1b03508ed47a1a0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d4519ada40b4b53a1b03508ed47a1a0, d80067ce5a344f35bf1996cf0f6e20c8; no voters:
I20260812 05:22:48.603806 21941 raft_consensus.cc:2781] T f81ecb65858549558f4ccda333a4a061 P 1d4519ada40b4b53a1b03508ed47a1a0 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 05:22:48.603958 21941 raft_consensus.cc:686] T f81ecb65858549558f4ccda333a4a061 P 1d4519ada40b4b53a1b03508ed47a1a0 [term 1 LEADER]: Becoming Leader. State: Replica: 1d4519ada40b4b53a1b03508ed47a1a0, State: Running, Role: LEADER
I20260812 05:22:48.604044 21941 consensus_queue.cc:237] T f81ecb65858549558f4ccda333a4a061 P 1d4519ada40b4b53a1b03508ed47a1a0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d80067ce5a344f35bf1996cf0f6e20c8" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 39845 } } peers { permanent_uuid: "1d4519ada40b4b53a1b03508ed47a1a0" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 36419 } } peers { permanent_uuid: "0502228f6e7340dea89eeaf2aa9f6e75" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 33025 } }
I20260812 05:22:48.604923 21081 catalog_manager.cc:5495] T f81ecb65858549558f4ccda333a4a061 P 1d4519ada40b4b53a1b03508ed47a1a0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1d4519ada40b4b53a1b03508ed47a1a0 (127.31.100.3). New cstate: current_term: 1 leader_uuid: "1d4519ada40b4b53a1b03508ed47a1a0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d80067ce5a344f35bf1996cf0f6e20c8" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 39845 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1d4519ada40b4b53a1b03508ed47a1a0" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 36419 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0502228f6e7340dea89eeaf2aa9f6e75" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 33025 } health_report { overall_health: UNKNOWN } } }
I20260812 05:22:48.605472 21943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:22:48.587876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.1:33025 (local address 127.31.100.3:34191)
0812 05:22:48.594189 (+ 6313us) negotiation.cc:107] Waiting for socket to connect
0812 05:22:48.594198 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0812 05:22:48.594271 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:22:48.594435 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:22:48.594435 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:22:48.594437 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 05:22:48.594563 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:22:48.594565 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:22:48.595452 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:22:48.595453 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:22:48.596840 (+ 1387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:22:48.596844 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:22:48.599631 (+ 2787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:22:48.599631 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:22:48.599753 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:22:48.599756 (+ 3us) client_negotiation.cc:769] Sending connection context
0812 05:22:48.600469 (+ 713us) client_negotiation.cc:240] Negotiation successful
0812 05:22:48.600511 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6295,"thread_start_us":115,"threads_started":1}
I20260812 05:22:48.606031 21330 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f81ecb65858549558f4ccda333a4a061" candidate_uuid: "1d4519ada40b4b53a1b03508ed47a1a0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0502228f6e7340dea89eeaf2aa9f6e75" is_pre_election: true
I20260812 05:22:48.606174 21330 raft_consensus.cc:2454] T f81ecb65858549558f4ccda333a4a061 P 0502228f6e7340dea89eeaf2aa9f6e75 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d4519ada40b4b53a1b03508ed47a1a0 in term 0.
I20260812 05:22:48.606281 21329 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f81ecb65858549558f4ccda333a4a061" candidate_uuid: "1d4519ada40b4b53a1b03508ed47a1a0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0502228f6e7340dea89eeaf2aa9f6e75"
I20260812 05:22:48.606333 21329 raft_consensus.cc:3037] T f81ecb65858549558f4ccda333a4a061 P 0502228f6e7340dea89eeaf2aa9f6e75 [term 0 FOLLOWER]: Advancing to term 1
I20260812 05:22:48.607524 21329 raft_consensus.cc:2454] T f81ecb65858549558f4ccda333a4a061 P 0502228f6e7340dea89eeaf2aa9f6e75 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d4519ada40b4b53a1b03508ed47a1a0 in term 1.
W20260812 05:22:48.611397 21874 tablet.cc:2369] T f81ecb65858549558f4ccda333a4a061 P 1d4519ada40b4b53a1b03508ed47a1a0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 05:22:48.636160 21918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:22:48.632775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47890 (local address 127.31.100.3:36419)
0812 05:22:48.632823 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0812 05:22:48.632825 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 05:22:48.632834 (+ 9us) server_negotiation.cc:407] Connection header received
0812 05:22:48.632873 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:22:48.632874 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:22:48.632895 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:22:48.632971 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 05:22:48.633184 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:22:48.633939 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:22:48.635061 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:22:48.635351 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:22:48.635431 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:22:48.635642 (+ 211us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 05:22:48.635774 (+ 132us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 05:22:48.635843 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0812 05:22:48.636062 (+ 219us) server_negotiation.cc:299] Negotiation successful
0812 05:22:48.636082 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260812 05:22:48.640082 21329 raft_consensus.cc:1261] T f81ecb65858549558f4ccda333a4a061 P 0502228f6e7340dea89eeaf2aa9f6e75 [term 1 FOLLOWER]: Refusing update from remote peer 1d4519ada40b4b53a1b03508ed47a1a0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 05:22:48.640544 21946 consensus_queue.cc:1035] T f81ecb65858549558f4ccda333a4a061 P 1d4519ada40b4b53a1b03508ed47a1a0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0502228f6e7340dea89eeaf2aa9f6e75" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 33025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 05:22:48.642812 21532 raft_consensus.cc:1261] T f81ecb65858549558f4ccda333a4a061 P d80067ce5a344f35bf1996cf0f6e20c8 [term 1 FOLLOWER]: Refusing update from remote peer 1d4519ada40b4b53a1b03508ed47a1a0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 05:22:48.643209 21946 consensus_queue.cc:1035] T f81ecb65858549558f4ccda333a4a061 P 1d4519ada40b4b53a1b03508ed47a1a0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d80067ce5a344f35bf1996cf0f6e20c8" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 39845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 05:22:48.646764 21949 mvcc.cc:205] Tried to move back new op lower bound from 7317553842746191872 to 7317553842602635264. Current Snapshot: MvccSnapshot[applied={T|T < 7317553842746191872}]
I20260812 05:22:48.647368 21953 mvcc.cc:205] Tried to move back new op lower bound from 7317553842746191872 to 7317553842602635264. Current Snapshot: MvccSnapshot[applied={T|T < 7317553842746191872}]
I20260812 05:22:48.649806 21081 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57938:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 05:22:48.649896 21081 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57938:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 05:22:48.652029 21081 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 04405e5fcf124400abdb1c71ecfb4985: Sending DeleteTablet for 3 replicas of tablet f81ecb65858549558f4ccda333a4a061
I20260812 05:22:48.652523 21274 tablet_service.cc:1514] Processing DeleteTablet for tablet f81ecb65858549558f4ccda333a4a061 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 05:22:48 UTC) from {username='kudu', principal='kudu/127.31.100.62@KRBTEST.COM'} at 127.0.0.1:45964
I20260812 05:22:48.652966 21522 tablet_service.cc:1514] Processing DeleteTablet for tablet f81ecb65858549558f4ccda333a4a061 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 05:22:48 UTC) from {username='kudu', principal='kudu/127.31.100.62@KRBTEST.COM'} at 127.0.0.1:50590
I20260812 05:22:48.653978 21714 tablet_service.cc:1514] Processing DeleteTablet for tablet f81ecb65858549558f4ccda333a4a061 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 05:22:48 UTC) from {username='kudu', principal='kudu/127.31.100.62@KRBTEST.COM'} at 127.0.0.1:47880
I20260812 05:22:48.654361 32144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0waFvT/build/release/bin/kudu with pid 21150
I20260812 05:22:48.658099 21966 tablet_replica.cc:330] T f81ecb65858549558f4ccda333a4a061 P 1d4519ada40b4b53a1b03508ed47a1a0: stopping tablet replica
I20260812 05:22:48.658233 21966 raft_consensus.cc:2229] T f81ecb65858549558f4ccda333a4a061 P 1d4519ada40b4b53a1b03508ed47a1a0 [term 1 LEADER]: Raft consensus shutting down.
I20260812 05:22:48.658340 21966 raft_consensus.cc:2258] T f81ecb65858549558f4ccda333a4a061 P 1d4519ada40b4b53a1b03508ed47a1a0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 05:22:48.677798 21966 ts_tablet_manager.cc:1905] T f81ecb65858549558f4ccda333a4a061 P 1d4519ada40b4b53a1b03508ed47a1a0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 05:22:48.679646 21966 ts_tablet_manager.cc:1918] T f81ecb65858549558f4ccda333a4a061 P 1d4519ada40b4b53a1b03508ed47a1a0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 05:22:48.679719 21966 log.cc:1196] T f81ecb65858549558f4ccda333a4a061 P 1d4519ada40b4b53a1b03508ed47a1a0: Deleting WAL directory at /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786512017613366-32144-0/minicluster-data/ts-2/wal/wals/f81ecb65858549558f4ccda333a4a061
I20260812 05:22:48.680002 21966 ts_tablet_manager.cc:1939] T f81ecb65858549558f4ccda333a4a061 P 1d4519ada40b4b53a1b03508ed47a1a0: Deleting consensus metadata
I20260812 05:22:48.680485 21059 catalog_manager.cc:4841] TS 1d4519ada40b4b53a1b03508ed47a1a0 (127.31.100.3:36419): tablet f81ecb65858549558f4ccda333a4a061 (table test-table [id=b883decdc1a146f6ac4f2d0a862c43c6]) successfully deleted
I20260812 05:22:48.680635 21965 tablet_replica.cc:330] T f81ecb65858549558f4ccda333a4a061 P d80067ce5a344f35bf1996cf0f6e20c8: stopping tablet replica
I20260812 05:22:48.680788 21965 raft_consensus.cc:2229] T f81ecb65858549558f4ccda333a4a061 P d80067ce5a344f35bf1996cf0f6e20c8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 05:22:48.680934 21965 raft_consensus.cc:2258] T f81ecb65858549558f4ccda333a4a061 P d80067ce5a344f35bf1996cf0f6e20c8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 05:22:48.685695 21965 ts_tablet_manager.cc:1905] T f81ecb65858549558f4ccda333a4a061 P d80067ce5a344f35bf1996cf0f6e20c8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 05:22:48.687037 21965 ts_tablet_manager.cc:1918] T f81ecb65858549558f4ccda333a4a061 P d80067ce5a344f35bf1996cf0f6e20c8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 05:22:48.687201 21965 log.cc:1196] T f81ecb65858549558f4ccda333a4a061 P d80067ce5a344f35bf1996cf0f6e20c8: Deleting WAL directory at /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786512017613366-32144-0/minicluster-data/ts-1/wal/wals/f81ecb65858549558f4ccda333a4a061
I20260812 05:22:48.687520 21965 ts_tablet_manager.cc:1939] T f81ecb65858549558f4ccda333a4a061 P d80067ce5a344f35bf1996cf0f6e20c8: Deleting consensus metadata
I20260812 05:22:48.688016 21055 catalog_manager.cc:4841] TS d80067ce5a344f35bf1996cf0f6e20c8 (127.31.100.2:39845): tablet f81ecb65858549558f4ccda333a4a061 (table test-table [id=b883decdc1a146f6ac4f2d0a862c43c6]) successfully deleted
W20260812 05:22:48.701927 21055 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.31.100.1:33025 (error 108)
I20260812 05:22:48.705639 32144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0waFvT/build/release/bin/kudu with pid 21425
I20260812 05:22:48.723677 21917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:22:48.723483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.1:33025 (local address 127.0.0.1:45980)
0812 05:22:48.723552 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0812 05:22:48.723625 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.100.1:33025: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260812 05:22:48.723879 21055 catalog_manager.cc:4584] TS 0502228f6e7340dea89eeaf2aa9f6e75 (127.31.100.1:33025): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f81ecb65858549558f4ccda333a4a061: Network error: Client connection negotiation failed: client connection to 127.31.100.1:33025: connect: Connection refused (error 111)
Aug 12 05:22:48 dist-test-slave-42z9 krb5kdc[19378](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512167, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.100.62@KRBTEST.COM for HTTP/127.31.100.20@KRBTEST.COM
I20260812 05:22:48.740715 32144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0waFvT/build/release/bin/kudu with pid 21629
I20260812 05:22:48.756150 32144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0waFvT/build/release/bin/kudu with pid 21023
I20260812 05:22:48.770764 32144 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260812 05:22:49.136727 32144 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260812 05:22:49.136847 32144 mini_ranger.cc:66] Stopping Ranger...
I20260812 05:22:49.491469 32144 mini_ranger.cc:68] Stopped Ranger
2026-08-12 05:22:49.491 UTC [19425] LOG: received smart shutdown request
2026-08-12 05:22:49.495 UTC [19425] LOG: background worker "logical replication launcher" (PID 19441) exited with exit code 1
2026-08-12 05:22:49.496 UTC [19429] LOG: shutting down
2026-08-12 05:22:49.636 UTC [19425] LOG: database system is shut down
2026-08-12T05:22:49Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (47065 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786512017613366-32144-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 05:22:49 dist-test-slave-42z9 krb5kdc[21990](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 05:22:49 dist-test-slave-42z9 krb5kdc[21990](info): set up 2 sockets
Aug 12 05:22:49 dist-test-slave-42z9 krb5kdc[21990](info): commencing operation
krb5kdc: starting...
W20260812 05:22:50.902328 32144 mini_kdc.cc:121] Time spent starting KDC: real 1.150s user 0.004s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786512017613366-32144-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786512017613366-32144-0/krb5kdc/test-user.keytab.
Aug 12 05:22:50 dist-test-slave-42z9 krb5kdc[21990](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512170, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T05:22:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T05:22:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.100.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.100.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.100.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.62.keytab.
Entry for principal kudu/127.31.100.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.62.keytab.
I20260812 05:22:51.014040 32144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0waFvT/build/release/bin/kudu
/tmp/dist-test-task0waFvT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786512017613366-32144-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786512017613366-32144-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786512017613366-32144-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786512017613366-32144-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.100.62:32833
--webserver_interface=127.31.100.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.62.keytab
--principal=kudu/127.31.100.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.100.20:46191
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.100.62:32833,127.31.100.61:33635,127.31.100.60:35959
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786512017613366-32144-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786512017613366-32144-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786512017613366-32144-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 05:22:51.094697 22009 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 05:22:51.094921 22009 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 05:22:51.094951 22009 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 05:22:51.096056 22009 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 05:22:51.096101 22009 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 05:22:51.096120 22009 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 05:22:51.096135 22009 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 05:22:51.096153 22009 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 05:22:51.097724 22009 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.100.20:46191
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786512017613366-32144-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786512017613366-32144-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.100.62:32833,127.31.100.61:33635,127.31.100.60:35959
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.62.keytab
--principal=kudu/127.31.100.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.100.62:32833
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786512017613366-32144-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.100.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22009
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786512017613366-32144-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 04:20:34 UTC on ccycloud.quasar-phqdqp.root.comops.site
build id 4581871
I20260812 05:22:51.097972 22009 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 05:22:51.098169 22009 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 05:22:51.102854 22019 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 05:22:51.102910 22017 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 05:22:51.102823 22022 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 05:22:51.102937 22009 server_base.cc:1034] running on GCE node
I20260812 05:22:51.103243 22009 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 05:22:51.103531 22009 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 05:22:51.104676 22009 hybrid_clock.cc:648] HybridClock initialized: now 1786512171104663 us; error 29 us; skew 500 ppm
Aug 12 05:22:51 dist-test-slave-42z9 krb5kdc[21990](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512171, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.100.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 05:22:51.110105 22009 init.cc:377] Logged in from keytab as kudu/127.31.100.62@KRBTEST.COM (short username kudu)
I20260812 05:22:51.110532 22009 webserver.cc:460] Webserver started at http://127.31.100.62:37851/ using document root <none> and password file <none>
I20260812 05:22:51.110705 22009 fs_manager.cc:359] Metadata directory not provided
I20260812 05:22:51.110742 22009 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 05:22:51.110831 22009 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 05:22:51.111786 22009 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786512017613366-32144-0/minicluster-data/master-0/data/instance:
uuid: "06d3c62f634b41a38fadb485b8ab0c77"
format_stamp: "Formatted at 2026-08-12 05:22:51 on dist-test-slave-42z9"
server_key: "b8e768fff2d179b4a17cb2e19295cb38"
server_key_iv: "903ae5c6f49b28a2f69a8f2af8b4279c"
server_key_version: "encryptionkey@0"
I20260812 05:22:51.112082 22009 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786512017613366-32144-0/minicluster-data/master-0/wal/instance:
uuid: "06d3c62f634b41a38fadb485b8ab0c77"
format_stamp: "Formatted at 2026-08-12 05:22:51 on dist-test-slave-42z9"
server_key: "b8e768fff2d179b4a17cb2e19295cb38"
server_key_iv: "903ae5c6f49b28a2f69a8f2af8b4279c"
server_key_version: "encryptionkey@0"
I20260812 05:22:51.113219 22009 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260812 05:22:51.113911 22034 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 05:22:51.114073 22009 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260812 05:22:51.114161 22009 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786512017613366-32144-0/minicluster-data/master-0/data,/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786512017613366-32144-0/minicluster-data/master-0/wal
uuid: "06d3c62f634b41a38fadb485b8ab0c77"
format_stamp: "Formatted at 2026-08-12 05:22:51 on dist-test-slave-42z9"
server_key: "b8e768fff2d179b4a17cb2e19295cb38"
server_key_iv: "903ae5c6f49b28a2f69a8f2af8b4279c"
server_key_version: "encryptionkey@0"
I20260812 05:22:51.114210 22009 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786512017613366-32144-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786512017613366-32144-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786512017613366-32144-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 05:22:51.147742 22009 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 05:22:51.149734 22009 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 05:22:51.149848 22009 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 05:22:51.155097 22009 rpc_server.cc:307] RPC server started. Bound to: 127.31.100.62:32833
I20260812 05:22:51.155154 22126 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.100.62:32833 every 8 connection(s)
I20260812 05:22:51.155665 22009 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786512017613366-32144-0/minicluster-data/master-0/data/info.pb
I20260812 05:22:51.156574 22128 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 05:22:51.158346 32144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0waFvT/build/release/bin/kudu as pid 22009
I20260812 05:22:51.158468 32144 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786512017613366-32144-0/minicluster-data/master-0/wal/instance
I20260812 05:22:51.158686 32144 external_mini_cluster.cc:1428] Setting key 92cd42d5d8fb539e8b5698cbb8bfe112
I20260812 05:22:51.158701 22128 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.100.62" port: 32833 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 05:22:51 dist-test-slave-42z9 krb5kdc[21990](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512171, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.100.62@KRBTEST.COM for kudu/127.31.100.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.31.100.61@KRBTEST.COM; defaulting to no policy
I20260812 05:22:51.168624 22135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:22:51.159410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.62:32833 (local address 127.0.0.1:45832)
0812 05:22:51.159598 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0812 05:22:51.159640 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0812 05:22:51.160205 (+ 565us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:22:51.160478 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:22:51.160481 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:22:51.160772 (+ 291us) client_negotiation.cc:189] Negotiated authn=SASL
0812 05:22:51.160929 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:22:51.160933 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:22:51.162223 (+ 1290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:22:51.162224 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:22:51.163284 (+ 1060us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:22:51.163287 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:22:51.163750 (+ 463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:22:51.163751 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:22:51.163821 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:22:51.164308 (+ 487us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 05:22:51.164316 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0812 05:22:51.165760 (+ 1444us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 05:22:51.167359 (+ 1599us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 05:22:51.167365 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 05:22:51.167367 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 05:22:51.167560 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 05:22:51.167797 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 05:22:51.167798 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 05:22:51.167798 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 05:22:51.167892 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 05:22:51.168197 (+ 305us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 05:22:51.168203 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 05:22:51.168272 (+ 69us) client_negotiation.cc:769] Sending connection context
0812 05:22:51.168340 (+ 68us) client_negotiation.cc:240] Negotiation successful
0812 05:22:51.168529 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"mutex_wait_us":67,"thread_start_us":85,"threads_started":1}
I20260812 05:22:51.168629 22136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:22:51.159462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45832 (local address 127.31.100.62:32833)
0812 05:22:51.159602 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0812 05:22:51.159611 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 05:22:51.159718 (+ 107us) server_negotiation.cc:407] Connection header received
0812 05:22:51.160339 (+ 621us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:22:51.160341 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:22:51.160359 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:22:51.160456 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:22:51.161027 (+ 571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:22:51.162128 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:22:51.163357 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:22:51.163666 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:22:51.163764 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:22:51.165841 (+ 2077us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 05:22:51.165848 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 05:22:51.165848 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 05:22:51.165854 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0812 05:22:51.167276 (+ 1422us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:22:51.167632 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:22:51.167634 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:22:51.167634 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:22:51.167737 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:22:51.167947 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:22:51.167948 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:22:51.167948 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:22:51.168136 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 05:22:51.168194 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0812 05:22:51.168367 (+ 173us) server_negotiation.cc:299] Negotiation successful
0812 05:22:51.168452 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108,"thread_start_us":58,"threads_started":1}
I20260812 05:22:51.169261 22128 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.100.61" port: 33635 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.100.61@KRBTEST.COM" created.
I20260812 05:22:51.169603 22135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:22:51.169499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.61:33635 (local address 127.0.0.1:46082)
0812 05:22:51.169540 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0812 05:22:51.169584 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.100.61:33635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260812 05:22:51.169732 22042 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.100.61:33635: connect: Connection refused (error 111)
I20260812 05:22:51.170096 22135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:22:51.170008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.61:33635 (local address 127.0.0.1:46098)
0812 05:22:51.170045 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0812 05:22:51.170075 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.100.61:33635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260812 05:22:51.170192 22128 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.100.61:33635: Network error: Client connection negotiation failed: client connection to 127.31.100.61:33635: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.100.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.61.keytab.
Entry for principal kudu/127.31.100.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.61.keytab.
I20260812 05:22:51.183683 32144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0waFvT/build/release/bin/kudu
/tmp/dist-test-task0waFvT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786512017613366-32144-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786512017613366-32144-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786512017613366-32144-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786512017613366-32144-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.100.61:33635
--webserver_interface=127.31.100.61
--webserver_port=0
--keytab_file=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.61.keytab
--principal=kudu/127.31.100.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.100.20:46191
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.100.62:32833,127.31.100.61:33635,127.31.100.60:35959
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786512017613366-32144-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786512017613366-32144-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786512017613366-32144-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 05:22:51.219398 22128 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.100.61" port: 33635 } attempt: 1
I20260812 05:22:51.219902 22135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:22:51.219769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.61:33635 (local address 127.0.0.1:46108)
0812 05:22:51.219830 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0812 05:22:51.219872 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.100.61:33635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260812 05:22:51.220279 22135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:22:51.220182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.61:33635 (local address 127.0.0.1:46112)
0812 05:22:51.220231 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0812 05:22:51.220256 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.100.61:33635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260812 05:22:51.220377 22128 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.100.61:33635: Network error: Client connection negotiation failed: client connection to 127.31.100.61:33635: connect: Connection refused (error 111)
W20260812 05:22:51.263614 22144 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 05:22:51.263806 22144 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 05:22:51.263836 22144 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 05:22:51.264919 22144 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 05:22:51.264958 22144 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 05:22:51.264976 22144 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 05:22:51.264992 22144 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 05:22:51.265007 22144 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 05:22:51.266487 22144 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.100.20:46191
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786512017613366-32144-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786512017613366-32144-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.31.100.62:32833,127.31.100.61:33635,127.31.100.60:35959
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.61.keytab
--principal=kudu/127.31.100.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.100.61:33635
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786512017613366-32144-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.31.100.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22144
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786512017613366-32144-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 04:20:34 UTC on ccycloud.quasar-phqdqp.root.comops.site
build id 4581871
I20260812 05:22:51.266732 22144 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 05:22:51.266928 22144 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 05:22:51.271481 22154 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 05:22:51.271497 22151 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 05:22:51.271481 22150 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 05:22:51.271687 22144 server_base.cc:1034] running on GCE node
I20260812 05:22:51.271935 22144 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 05:22:51.272207 22144 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 05:22:51.273372 22144 hybrid_clock.cc:648] HybridClock initialized: now 1786512171273353 us; error 30 us; skew 500 ppm
Aug 12 05:22:51 dist-test-slave-42z9 krb5kdc[21990](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512171, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.100.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 05:22:51.275861 22144 init.cc:377] Logged in from keytab as kudu/127.31.100.61@KRBTEST.COM (short username kudu)
I20260812 05:22:51.276253 22144 webserver.cc:460] Webserver started at http://127.31.100.61:46299/ using document root <none> and password file <none>
I20260812 05:22:51.276443 22144 fs_manager.cc:359] Metadata directory not provided
I20260812 05:22:51.276481 22144 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 05:22:51.276582 22144 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 05:22:51.277617 22144 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786512017613366-32144-0/minicluster-data/master-1/data/instance:
uuid: "8229f066d5604ad6b94fb1078340f112"
format_stamp: "Formatted at 2026-08-12 05:22:51 on dist-test-slave-42z9"
server_key: "9011ee2bb817cb1d360ac523af7243ff"
server_key_iv: "c29aa8167648f3d6bcb9bf6d3314da8e"
server_key_version: "encryptionkey@0"
I20260812 05:22:51.277923 22144 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786512017613366-32144-0/minicluster-data/master-1/wal/instance:
uuid: "8229f066d5604ad6b94fb1078340f112"
format_stamp: "Formatted at 2026-08-12 05:22:51 on dist-test-slave-42z9"
server_key: "9011ee2bb817cb1d360ac523af7243ff"
server_key_iv: "c29aa8167648f3d6bcb9bf6d3314da8e"
server_key_version: "encryptionkey@0"
I20260812 05:22:51.279240 22144 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260812 05:22:51.279953 22163 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 05:22:51.280126 22144 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260812 05:22:51.280205 22144 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786512017613366-32144-0/minicluster-data/master-1/data,/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786512017613366-32144-0/minicluster-data/master-1/wal
uuid: "8229f066d5604ad6b94fb1078340f112"
format_stamp: "Formatted at 2026-08-12 05:22:51 on dist-test-slave-42z9"
server_key: "9011ee2bb817cb1d360ac523af7243ff"
server_key_iv: "c29aa8167648f3d6bcb9bf6d3314da8e"
server_key_version: "encryptionkey@0"
I20260812 05:22:51.280257 22144 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786512017613366-32144-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786512017613366-32144-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786512017613366-32144-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 05:22:51.288538 22128 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.100.61" port: 33635 } attempt: 2
I20260812 05:22:51.288910 22135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:22:51.288796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.61:33635 (local address 127.0.0.1:46124)
0812 05:22:51.288849 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0812 05:22:51.288883 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.100.61:33635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260812 05:22:51.289227 22135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:22:51.289148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.61:33635 (local address 127.0.0.1:46126)
0812 05:22:51.289186 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0812 05:22:51.289209 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.100.61:33635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260812 05:22:51.289309 22128 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.100.61:33635: Network error: Client connection negotiation failed: client connection to 127.31.100.61:33635: connect: Connection refused (error 111)
I20260812 05:22:51.290025 22144 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 05:22:51.291922 22144 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 05:22:51.292033 22144 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 05:22:51.295748 22144 rpc_server.cc:307] RPC server started. Bound to: 127.31.100.61:33635
I20260812 05:22:51.295814 22246 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.100.61:33635 every 8 connection(s)
I20260812 05:22:51.296144 22144 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786512017613366-32144-0/minicluster-data/master-1/data/info.pb
I20260812 05:22:51.296979 22247 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 05:22:51.298359 32144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0waFvT/build/release/bin/kudu as pid 22144
I20260812 05:22:51.298475 32144 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786512017613366-32144-0/minicluster-data/master-1/wal/instance
I20260812 05:22:51.298666 32144 external_mini_cluster.cc:1428] Setting key ba3bc401923de1371c20ef09855869d5
I20260812 05:22:51.298684 22247 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.100.62" port: 32833 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 05:22:51 dist-test-slave-42z9 krb5kdc[21990](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512171, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.100.61@KRBTEST.COM for kudu/127.31.100.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.31.100.60@KRBTEST.COM; defaulting to no policy
I20260812 05:22:51.307716 22136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:22:51.299399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45838 (local address 127.31.100.62:32833)
0812 05:22:51.300534 (+ 1135us) server_negotiation.cc:206] Beginning negotiation
0812 05:22:51.300535 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0812 05:22:51.300543 (+ 8us) server_negotiation.cc:407] Connection header received
0812 05:22:51.300564 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:22:51.300565 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:22:51.300573 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:22:51.300623 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:22:51.301223 (+ 600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:22:51.301766 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:22:51.302756 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:22:51.302971 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:22:51.303029 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:22:51.305114 (+ 2085us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 05:22:51.305115 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 05:22:51.305116 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 05:22:51.305122 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0812 05:22:51.306580 (+ 1458us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:22:51.306927 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:22:51.306928 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:22:51.306928 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:22:51.306974 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:22:51.307196 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:22:51.307197 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:22:51.307197 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:22:51.307326 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 05:22:51.307378 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0812 05:22:51.307582 (+ 204us) server_negotiation.cc:299] Negotiation successful
0812 05:22:51.307646 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1110}
I20260812 05:22:51.307849 22249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:22:51.299342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.62:32833 (local address 127.0.0.1:45838)
0812 05:22:51.299534 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0812 05:22:51.299579 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0812 05:22:51.300092 (+ 513us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:22:51.300715 (+ 623us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:22:51.300719 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:22:51.300943 (+ 224us) client_negotiation.cc:189] Negotiated authn=SASL
0812 05:22:51.301128 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:22:51.301132 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:22:51.301851 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:22:51.301852 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:22:51.302677 (+ 825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:22:51.302680 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:22:51.303046 (+ 366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:22:51.303047 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:22:51.303162 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:22:51.303608 (+ 446us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 05:22:51.303616 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0812 05:22:51.304997 (+ 1381us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 05:22:51.306654 (+ 1657us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 05:22:51.306661 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 05:22:51.306662 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 05:22:51.306861 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 05:22:51.307030 (+ 169us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 05:22:51.307030 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 05:22:51.307031 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 05:22:51.307123 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 05:22:51.307377 (+ 254us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 05:22:51.307380 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 05:22:51.307482 (+ 102us) client_negotiation.cc:769] Sending connection context
0812 05:22:51.307560 (+ 78us) client_negotiation.cc:240] Negotiation successful
0812 05:22:51.307672 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":94,"threads_started":1}
I20260812 05:22:51.308333 22247 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.100.61" port: 33635 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.100.60@KRBTEST.COM" created.
Aug 12 05:22:51 dist-test-slave-42z9 krb5kdc[21990](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512171, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.100.61@KRBTEST.COM for kudu/127.31.100.61@KRBTEST.COM
I20260812 05:22:51.315582 22249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:22:51.308567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.61:33635 (local address 127.0.0.1:46136)
0812 05:22:51.308721 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0812 05:22:51.308730 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0812 05:22:51.308768 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:22:51.309666 (+ 898us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:22:51.309667 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:22:51.309793 (+ 126us) client_negotiation.cc:189] Negotiated authn=SASL
0812 05:22:51.309837 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:22:51.309838 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:22:51.310639 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:22:51.310640 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:22:51.311157 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:22:51.311159 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:22:51.311574 (+ 415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:22:51.311575 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:22:51.311632 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:22:51.311949 (+ 317us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 05:22:51.311953 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0812 05:22:51.313237 (+ 1284us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 05:22:51.314687 (+ 1450us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 05:22:51.314688 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 05:22:51.314688 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 05:22:51.314834 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 05:22:51.315007 (+ 173us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 05:22:51.315008 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 05:22:51.315008 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 05:22:51.315060 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 05:22:51.315296 (+ 236us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 05:22:51.315296 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 05:22:51.315332 (+ 36us) client_negotiation.cc:769] Sending connection context
0812 05:22:51.315393 (+ 61us) client_negotiation.cc:240] Negotiation successful
0812 05:22:51.315483 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141}
I20260812 05:22:51.315665 22251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:22:51.308599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46136 (local address 127.31.100.61:33635)
0812 05:22:51.309524 (+ 925us) server_negotiation.cc:206] Beginning negotiation
0812 05:22:51.309529 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 05:22:51.309553 (+ 24us) server_negotiation.cc:407] Connection header received
0812 05:22:51.309573 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:22:51.309575 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:22:51.309588 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:22:51.309657 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:22:51.309902 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:22:51.310559 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:22:51.311223 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:22:51.311508 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:22:51.311565 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:22:51.313311 (+ 1746us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 05:22:51.313318 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 05:22:51.313318 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 05:22:51.313325 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 05:22:51.314613 (+ 1288us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:22:51.314901 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:22:51.314903 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:22:51.314904 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:22:51.314954 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:22:51.315111 (+ 157us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:22:51.315112 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:22:51.315112 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:22:51.315237 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 05:22:51.315303 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0812 05:22:51.315430 (+ 127us) server_negotiation.cc:299] Negotiation successful
0812 05:22:51.315551 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":42,"server-negotiator.queue_time_us":895,"thread_start_us":81,"threads_started":1}
I20260812 05:22:51.316130 22247 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.100.60" port: 35959 } has no permanent_uuid. Determining permanent_uuid...
I20260812 05:22:51.316416 22249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:22:51.316323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.60:35959 (local address 127.0.0.1:46378)
0812 05:22:51.316353 (+ 30us) negotiation.cc:107] Waiting for socket to connect
0812 05:22:51.316397 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.100.60:35959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":16}
W20260812 05:22:51.316548 22164 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.100.60:35959: connect: Connection refused (error 111)
I20260812 05:22:51.316917 22249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:22:51.316833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.60:35959 (local address 127.0.0.1:46384)
0812 05:22:51.316872 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0812 05:22:51.316901 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.100.60:35959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260812 05:22:51.317010 22247 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.100.60:35959: Network error: Client connection negotiation failed: client connection to 127.31.100.60:35959: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.100.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.60.keytab.
Entry for principal kudu/127.31.100.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.60.keytab.
I20260812 05:22:51.321619 32144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0waFvT/build/release/bin/kudu
/tmp/dist-test-task0waFvT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786512017613366-32144-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786512017613366-32144-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786512017613366-32144-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786512017613366-32144-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.100.60:35959
--webserver_interface=127.31.100.60
--webserver_port=0
--keytab_file=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.60.keytab
--principal=kudu/127.31.100.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.100.20:46191
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.100.62:32833,127.31.100.61:33635,127.31.100.60:35959
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786512017613366-32144-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786512017613366-32144-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786512017613366-32144-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 05:22:51.366228 22247 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.100.60" port: 35959 } attempt: 1
I20260812 05:22:51.366756 22249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:22:51.366616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.60:35959 (local address 127.0.0.1:46388)
0812 05:22:51.366684 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0812 05:22:51.366725 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.100.60:35959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260812 05:22:51.367090 22249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:22:51.367006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.60:35959 (local address 127.0.0.1:46392)
0812 05:22:51.367047 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0812 05:22:51.367071 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.100.60:35959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260812 05:22:51.367192 22247 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.100.60:35959: Network error: Client connection negotiation failed: client connection to 127.31.100.60:35959: connect: Connection refused (error 111)
I20260812 05:22:51.380468 22128 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.100.61" port: 33635 } attempt: 3
Aug 12 05:22:51 dist-test-slave-42z9 krb5kdc[21990](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512171, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.100.62@KRBTEST.COM for kudu/127.31.100.61@KRBTEST.COM
I20260812 05:22:51.387454 22135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:22:51.380791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.61:33635 (local address 127.0.0.1:46146)
0812 05:22:51.380852 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0812 05:22:51.380873 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0812 05:22:51.380936 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:22:51.381080 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:22:51.381080 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:22:51.381276 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0812 05:22:51.381360 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:22:51.381363 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:22:51.382059 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:22:51.382060 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:22:51.382898 (+ 838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:22:51.382902 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:22:51.383262 (+ 360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:22:51.383263 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:22:51.383343 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:22:51.383761 (+ 418us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 05:22:51.383767 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0812 05:22:51.385096 (+ 1329us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 05:22:51.386544 (+ 1448us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 05:22:51.386545 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 05:22:51.386545 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 05:22:51.386707 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 05:22:51.386881 (+ 174us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 05:22:51.386882 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 05:22:51.386882 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 05:22:51.386937 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 05:22:51.387149 (+ 212us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 05:22:51.387150 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 05:22:51.387196 (+ 46us) client_negotiation.cc:769] Sending connection context
0812 05:22:51.387257 (+ 61us) client_negotiation.cc:240] Negotiation successful
0812 05:22:51.387356 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260812 05:22:51.387452 22251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:22:51.380857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46146 (local address 127.31.100.61:33635)
0812 05:22:51.380905 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0812 05:22:51.380907 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 05:22:51.380919 (+ 12us) server_negotiation.cc:407] Connection header received
0812 05:22:51.380996 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:22:51.380997 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:22:51.381008 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:22:51.381059 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:22:51.381426 (+ 367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:22:51.381975 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:22:51.382972 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:22:51.383191 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:22:51.383241 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:22:51.385175 (+ 1934us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 05:22:51.385176 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 05:22:51.385176 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 05:22:51.385184 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0812 05:22:51.386463 (+ 1279us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:22:51.386771 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:22:51.386772 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:22:51.386772 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:22:51.386825 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:22:51.386987 (+ 162us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:22:51.386987 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:22:51.386988 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:22:51.387092 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 05:22:51.387147 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0812 05:22:51.387289 (+ 142us) server_negotiation.cc:299] Negotiation successful
0812 05:22:51.387358 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260812 05:22:51.387815 22128 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.100.60" port: 35959 } has no permanent_uuid. Determining permanent_uuid...
I20260812 05:22:51.388137 22135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:22:51.388042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.60:35959 (local address 127.0.0.1:46408)
0812 05:22:51.388087 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0812 05:22:51.388113 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.100.60:35959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260812 05:22:51.388471 22135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:22:51.388393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.60:35959 (local address 127.0.0.1:46410)
0812 05:22:51.388435 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0812 05:22:51.388457 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.100.60:35959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260812 05:22:51.388556 22128 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.100.60:35959: Network error: Client connection negotiation failed: client connection to 127.31.100.60:35959: connect: Connection refused (error 111)
W20260812 05:22:51.403611 22255 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 05:22:51.403795 22255 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 05:22:51.403813 22255 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 05:22:51.404949 22255 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 05:22:51.404991 22255 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 05:22:51.405005 22255 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 05:22:51.405025 22255 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 05:22:51.405048 22255 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 05:22:51.406636 22255 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.100.20:46191
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786512017613366-32144-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786512017613366-32144-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.31.100.62:32833,127.31.100.61:33635,127.31.100.60:35959
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.60.keytab
--principal=kudu/127.31.100.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.100.60:35959
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786512017613366-32144-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.31.100.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22255
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786512017613366-32144-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 04:20:34 UTC on ccycloud.quasar-phqdqp.root.comops.site
build id 4581871
I20260812 05:22:51.406867 22255 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 05:22:51.407084 22255 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 05:22:51.411988 22264 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 05:22:51.412024 22255 server_base.cc:1034] running on GCE node
W20260812 05:22:51.411995 22263 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 05:22:51.412003 22266 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 05:22:51.412408 22255 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 05:22:51.412693 22255 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 05:22:51.413829 22255 hybrid_clock.cc:648] HybridClock initialized: now 1786512171413812 us; error 33 us; skew 500 ppm
Aug 12 05:22:51 dist-test-slave-42z9 krb5kdc[21990](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512171, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.100.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 05:22:51.416236 22255 init.cc:377] Logged in from keytab as kudu/127.31.100.60@KRBTEST.COM (short username kudu)
I20260812 05:22:51.416632 22255 webserver.cc:460] Webserver started at http://127.31.100.60:42761/ using document root <none> and password file <none>
I20260812 05:22:51.416869 22255 fs_manager.cc:359] Metadata directory not provided
I20260812 05:22:51.416929 22255 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 05:22:51.417057 22255 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 05:22:51.418156 22255 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786512017613366-32144-0/minicluster-data/master-2/data/instance:
uuid: "bc3fd4361d7f4d16b743b7cef31d5779"
format_stamp: "Formatted at 2026-08-12 05:22:51 on dist-test-slave-42z9"
server_key: "8b93fc38a4f35baa512cd58ca6b03168"
server_key_iv: "198400d69a0097abc69d93ffbd93b96b"
server_key_version: "encryptionkey@0"
I20260812 05:22:51.418473 22255 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786512017613366-32144-0/minicluster-data/master-2/wal/instance:
uuid: "bc3fd4361d7f4d16b743b7cef31d5779"
format_stamp: "Formatted at 2026-08-12 05:22:51 on dist-test-slave-42z9"
server_key: "8b93fc38a4f35baa512cd58ca6b03168"
server_key_iv: "198400d69a0097abc69d93ffbd93b96b"
server_key_version: "encryptionkey@0"
I20260812 05:22:51.419682 22128 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.100.60" port: 35959 } attempt: 1
I20260812 05:22:51.419817 22255 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260812 05:22:51.420061 22135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:22:51.419934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.60:35959 (local address 127.0.0.1:46426)
0812 05:22:51.419999 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0812 05:22:51.420028 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.100.60:35959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260812 05:22:51.420414 22135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:22:51.420305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.60:35959 (local address 127.0.0.1:46438)
0812 05:22:51.420360 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0812 05:22:51.420390 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.100.60:35959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260812 05:22:51.420513 22128 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.100.60:35959: Network error: Client connection negotiation failed: client connection to 127.31.100.60:35959: connect: Connection refused (error 111)
I20260812 05:22:51.420608 22278 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 05:22:51.420769 22255 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260812 05:22:51.420854 22255 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786512017613366-32144-0/minicluster-data/master-2/data,/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786512017613366-32144-0/minicluster-data/master-2/wal
uuid: "bc3fd4361d7f4d16b743b7cef31d5779"
format_stamp: "Formatted at 2026-08-12 05:22:51 on dist-test-slave-42z9"
server_key: "8b93fc38a4f35baa512cd58ca6b03168"
server_key_iv: "198400d69a0097abc69d93ffbd93b96b"
server_key_version: "encryptionkey@0"
I20260812 05:22:51.420922 22255 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786512017613366-32144-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786512017613366-32144-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786512017613366-32144-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 05:22:51.435331 22247 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.100.60" port: 35959 } attempt: 2
I20260812 05:22:51.435757 22249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:22:51.435636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.60:35959 (local address 127.0.0.1:46442)
0812 05:22:51.435692 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0812 05:22:51.435730 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.100.60:35959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260812 05:22:51.436129 22249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:22:51.436037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.60:35959 (local address 127.0.0.1:46448)
0812 05:22:51.436075 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0812 05:22:51.436103 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.100.60:35959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260812 05:22:51.436218 22247 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.100.60:35959: Network error: Client connection negotiation failed: client connection to 127.31.100.60:35959: connect: Connection refused (error 111)
I20260812 05:22:51.439079 22255 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 05:22:51.440889 22255 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 05:22:51.440995 22255 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 05:22:51.444674 22255 rpc_server.cc:307] RPC server started. Bound to: 127.31.100.60:35959
I20260812 05:22:51.444720 22356 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.100.60:35959 every 8 connection(s)
I20260812 05:22:51.445043 22255 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786512017613366-32144-0/minicluster-data/master-2/data/info.pb
I20260812 05:22:51.445916 22357 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 05:22:51.446154 32144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0waFvT/build/release/bin/kudu as pid 22255
I20260812 05:22:51.446254 32144 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786512017613366-32144-0/minicluster-data/master-2/wal/instance
I20260812 05:22:51.446436 32144 external_mini_cluster.cc:1428] Setting key a1b9d6128ed971807b06ffa68c9a1b42
I20260812 05:22:51.447618 22357 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.100.62" port: 32833 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 05:22:51 dist-test-slave-42z9 krb5kdc[21990](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512171, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.100.60@KRBTEST.COM for kudu/127.31.100.62@KRBTEST.COM
Aug 12 05:22:51 dist-test-slave-42z9 krb5kdc[21990](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512170, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.100.60@KRBTEST.COM
I20260812 05:22:51.455441 22136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:22:51.448242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45842 (local address 127.31.100.62:32833)
0812 05:22:51.448345 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0812 05:22:51.448347 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 05:22:51.448394 (+ 47us) server_negotiation.cc:407] Connection header received
0812 05:22:51.448478 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:22:51.448479 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:22:51.448488 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:22:51.448530 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:22:51.449014 (+ 484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:22:51.449670 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:22:51.450627 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:22:51.450882 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:22:51.450956 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:22:51.452851 (+ 1895us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 05:22:51.452853 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 05:22:51.452854 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 05:22:51.452861 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 05:22:51.454137 (+ 1276us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:22:51.454538 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:22:51.454540 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:22:51.454540 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:22:51.454599 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:22:51.454805 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:22:51.454806 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:22:51.454807 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:22:51.454952 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 05:22:51.455016 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0812 05:22:51.455264 (+ 248us) server_negotiation.cc:299] Negotiation successful
0812 05:22:51.455348 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260812 05:22:51.455556 22362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:22:51.448163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.62:32833 (local address 127.0.0.1:45842)
0812 05:22:51.448305 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0812 05:22:51.448343 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0812 05:22:51.448412 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:22:51.448560 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:22:51.448562 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:22:51.448813 (+ 251us) client_negotiation.cc:189] Negotiated authn=SASL
0812 05:22:51.448921 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:22:51.448926 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:22:51.449748 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:22:51.449749 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:22:51.450544 (+ 795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:22:51.450546 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:22:51.450971 (+ 425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:22:51.450973 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:22:51.451050 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:22:51.451394 (+ 344us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 05:22:51.451401 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0812 05:22:51.452753 (+ 1352us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 05:22:51.454204 (+ 1451us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 05:22:51.454210 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 05:22:51.454212 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 05:22:51.454467 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 05:22:51.454663 (+ 196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 05:22:51.454663 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 05:22:51.454663 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 05:22:51.454750 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 05:22:51.455013 (+ 263us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 05:22:51.455016 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 05:22:51.455128 (+ 112us) client_negotiation.cc:769] Sending connection context
0812 05:22:51.455211 (+ 83us) client_negotiation.cc:240] Negotiation successful
0812 05:22:51.455351 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":70,"threads_started":1}
I20260812 05:22:51.456117 22360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:22:51.446882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46454 (local address 127.31.100.60:35959)
0812 05:22:51.447070 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0812 05:22:51.447078 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 05:22:51.447102 (+ 24us) server_negotiation.cc:407] Connection header received
0812 05:22:51.447506 (+ 404us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:22:51.447512 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:22:51.447565 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:22:51.447685 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:22:51.448323 (+ 638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:22:51.449175 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:22:51.450067 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:22:51.450632 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:22:51.450711 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:22:51.453273 (+ 2562us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 05:22:51.453280 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 05:22:51.453281 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 05:22:51.453286 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0812 05:22:51.455038 (+ 1752us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:22:51.455325 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:22:51.455328 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:22:51.455328 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:22:51.455398 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:22:51.455593 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:22:51.455595 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:22:51.455595 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:22:51.455755 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 05:22:51.455901 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0812 05:22:51.455953 (+ 52us) server_negotiation.cc:299] Negotiation successful
0812 05:22:51.456029 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":91,"threads_started":1}
I20260812 05:22:51.456424 22357 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.100.61" port: 33635 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 05:22:51 dist-test-slave-42z9 krb5kdc[21990](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512171, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.100.60@KRBTEST.COM for kudu/127.31.100.61@KRBTEST.COM
Aug 12 05:22:51 dist-test-slave-42z9 krb5kdc[21990](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512170, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.100.62@KRBTEST.COM
I20260812 05:22:51.462843 22362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:22:51.456659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.61:33635 (local address 127.0.0.1:46158)
0812 05:22:51.456705 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0812 05:22:51.456714 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0812 05:22:51.456750 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:22:51.456915 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:22:51.456916 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:22:51.457066 (+ 150us) client_negotiation.cc:189] Negotiated authn=SASL
0812 05:22:51.457096 (+ 30us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:22:51.457097 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:22:51.457842 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:22:51.457842 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:22:51.458390 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:22:51.458392 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:22:51.458798 (+ 406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:22:51.458798 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:22:51.458854 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:22:51.459206 (+ 352us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 05:22:51.459210 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0812 05:22:51.460397 (+ 1187us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 05:22:51.461957 (+ 1560us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 05:22:51.461958 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 05:22:51.461959 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 05:22:51.462151 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 05:22:51.462325 (+ 174us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 05:22:51.462326 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 05:22:51.462327 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 05:22:51.462397 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 05:22:51.462623 (+ 226us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 05:22:51.462624 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 05:22:51.462659 (+ 35us) client_negotiation.cc:769] Sending connection context
0812 05:22:51.462711 (+ 52us) client_negotiation.cc:240] Negotiation successful
0812 05:22:51.462776 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260812 05:22:51.462908 22251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:22:51.456732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46158 (local address 127.31.100.61:33635)
0812 05:22:51.456804 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0812 05:22:51.456806 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 05:22:51.456815 (+ 9us) server_negotiation.cc:407] Connection header received
0812 05:22:51.456839 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:22:51.456840 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:22:51.456850 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:22:51.456904 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:22:51.457172 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:22:51.457779 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:22:51.458461 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:22:51.458737 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:22:51.458795 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:22:51.460473 (+ 1678us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 05:22:51.460474 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 05:22:51.460475 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 05:22:51.460481 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0812 05:22:51.461876 (+ 1395us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:22:51.462218 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:22:51.462219 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:22:51.462219 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:22:51.462267 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:22:51.462455 (+ 188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:22:51.462455 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:22:51.462455 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:22:51.462559 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 05:22:51.462619 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0812 05:22:51.462751 (+ 132us) server_negotiation.cc:299] Negotiation successful
0812 05:22:51.462819 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260812 05:22:51.463243 22357 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.100.60" port: 35959 } has no permanent_uuid. Determining permanent_uuid...
I20260812 05:22:51.463873 22136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:22:51.456871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45856 (local address 127.31.100.62:32833)
0812 05:22:51.456920 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0812 05:22:51.456921 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0812 05:22:51.456928 (+ 7us) server_negotiation.cc:407] Connection header received
0812 05:22:51.456946 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:22:51.456947 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:22:51.456955 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:22:51.457003 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:22:51.457527 (+ 524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:22:51.458023 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:22:51.458748 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:22:51.458999 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:22:51.459059 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:22:51.461058 (+ 1999us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 05:22:51.461060 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 05:22:51.461060 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 05:22:51.461066 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0812 05:22:51.462518 (+ 1452us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:22:51.462859 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:22:51.462860 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:22:51.462860 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:22:51.462924 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:22:51.463119 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:22:51.463120 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:22:51.463120 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:22:51.463316 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 05:22:51.463469 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0812 05:22:51.463744 (+ 275us) server_negotiation.cc:299] Negotiation successful
0812 05:22:51.463805 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
Aug 12 05:22:51 dist-test-slave-42z9 krb5kdc[21990](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512171, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.100.60@KRBTEST.COM for kudu/127.31.100.60@KRBTEST.COM
Aug 12 05:22:51 dist-test-slave-42z9 krb5kdc[21990](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512170, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.100.61@KRBTEST.COM
I20260812 05:22:51.469684 22360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:22:51.463529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46456 (local address 127.31.100.60:35959)
0812 05:22:51.463561 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0812 05:22:51.463562 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0812 05:22:51.463569 (+ 7us) server_negotiation.cc:407] Connection header received
0812 05:22:51.463620 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:22:51.463621 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:22:51.463627 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:22:51.463666 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:22:51.463916 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:22:51.464544 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:22:51.465192 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:22:51.465614 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:22:51.465694 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:22:51.467405 (+ 1711us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 05:22:51.467406 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 05:22:51.467406 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 05:22:51.467412 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0812 05:22:51.468583 (+ 1171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:22:51.468893 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:22:51.468894 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:22:51.468894 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:22:51.468948 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:22:51.469125 (+ 177us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:22:51.469126 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:22:51.469126 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:22:51.469239 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 05:22:51.469299 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0812 05:22:51.469484 (+ 185us) server_negotiation.cc:299] Negotiation successful
0812 05:22:51.469566 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260812 05:22:51.469723 22362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:22:51.463485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.60:35959 (local address 127.0.0.1:46456)
0812 05:22:51.463527 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0812 05:22:51.463536 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0812 05:22:51.463571 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:22:51.463670 (+ 99us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:22:51.463671 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:22:51.463814 (+ 143us) client_negotiation.cc:189] Negotiated authn=SASL
0812 05:22:51.463845 (+ 31us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:22:51.463847 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:22:51.464597 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:22:51.464598 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:22:51.465119 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:22:51.465121 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:22:51.465693 (+ 572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:22:51.465694 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:22:51.465732 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:22:51.466050 (+ 318us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 05:22:51.466055 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0812 05:22:51.467324 (+ 1269us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 05:22:51.468658 (+ 1334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 05:22:51.468659 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 05:22:51.468660 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 05:22:51.468826 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 05:22:51.469013 (+ 187us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 05:22:51.469014 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 05:22:51.469014 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 05:22:51.469069 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 05:22:51.469300 (+ 231us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 05:22:51.469300 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 05:22:51.469342 (+ 42us) client_negotiation.cc:769] Sending connection context
0812 05:22:51.469407 (+ 65us) client_negotiation.cc:240] Negotiation successful
0812 05:22:51.469616 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28,"mutex_wait_us":57}
I20260812 05:22:51.471364 22251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:22:51.464410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46170 (local address 127.31.100.61:33635)
0812 05:22:51.464442 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0812 05:22:51.464444 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 05:22:51.464451 (+ 7us) server_negotiation.cc:407] Connection header received
0812 05:22:51.464497 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:22:51.464497 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:22:51.464504 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:22:51.464540 (+ 36us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:22:51.464974 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:22:51.465515 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:22:51.466189 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:22:51.466464 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:22:51.466534 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:22:51.468592 (+ 2058us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 05:22:51.468593 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 05:22:51.468593 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 05:22:51.468599 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0812 05:22:51.470383 (+ 1784us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:22:51.470686 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:22:51.470687 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:22:51.470687 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:22:51.470744 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:22:51.470884 (+ 140us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:22:51.470884 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:22:51.470885 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:22:51.471009 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 05:22:51.471176 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0812 05:22:51.471215 (+ 39us) server_negotiation.cc:299] Negotiation successful
0812 05:22:51.471283 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260812 05:22:51.471581 22357 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bc3fd4361d7f4d16b743b7cef31d5779: Bootstrap starting.
I20260812 05:22:51.472237 22357 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bc3fd4361d7f4d16b743b7cef31d5779: Neither blocks nor log segments found. Creating new log.
I20260812 05:22:51.472514 22357 log.cc:824] T 00000000000000000000000000000000 P bc3fd4361d7f4d16b743b7cef31d5779: Log is configured to *not* fsync() on all Append() calls
I20260812 05:22:51.473199 22357 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bc3fd4361d7f4d16b743b7cef31d5779: No bootstrap required, opened a new log
I20260812 05:22:51.473248 32144 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 05:22:51.474795 22357 raft_consensus.cc:348] T 00000000000000000000000000000000 P bc3fd4361d7f4d16b743b7cef31d5779 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06d3c62f634b41a38fadb485b8ab0c77" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 32833 } } peers { permanent_uuid: "8229f066d5604ad6b94fb1078340f112" member_type: VOTER last_known_addr { host: "127.31.100.61" port: 33635 } } peers { permanent_uuid: "bc3fd4361d7f4d16b743b7cef31d5779" member_type: VOTER last_known_addr { host: "127.31.100.60" port: 35959 } }
I20260812 05:22:51.474961 22357 raft_consensus.cc:374] T 00000000000000000000000000000000 P bc3fd4361d7f4d16b743b7cef31d5779 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 05:22:51.475013 22357 raft_consensus.cc:729] T 00000000000000000000000000000000 P bc3fd4361d7f4d16b743b7cef31d5779 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc3fd4361d7f4d16b743b7cef31d5779, State: Initialized, Role: FOLLOWER
I20260812 05:22:51.475152 22357 consensus_queue.cc:260] T 00000000000000000000000000000000 P bc3fd4361d7f4d16b743b7cef31d5779 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06d3c62f634b41a38fadb485b8ab0c77" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 32833 } } peers { permanent_uuid: "8229f066d5604ad6b94fb1078340f112" member_type: VOTER last_known_addr { host: "127.31.100.61" port: 33635 } } peers { permanent_uuid: "bc3fd4361d7f4d16b743b7cef31d5779" member_type: VOTER last_known_addr { host: "127.31.100.60" port: 35959 } }
I20260812 05:22:51.475469 22357 sys_catalog.cc:564] T 00000000000000000000000000000000 P bc3fd4361d7f4d16b743b7cef31d5779 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 05:22:51.478148 22374 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc3fd4361d7f4d16b743b7cef31d5779 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06d3c62f634b41a38fadb485b8ab0c77" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 32833 } } peers { permanent_uuid: "8229f066d5604ad6b94fb1078340f112" member_type: VOTER last_known_addr { host: "127.31.100.61" port: 33635 } } peers { permanent_uuid: "bc3fd4361d7f4d16b743b7cef31d5779" member_type: VOTER last_known_addr { host: "127.31.100.60" port: 35959 } } }
I20260812 05:22:51.478226 22374 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc3fd4361d7f4d16b743b7cef31d5779 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 05:22:51.477344 22255 master_runner.cc:186] Error getting master registration for 127.31.100.62:32833: OK, instance_id { permanent_uuid: "06d3c62f634b41a38fadb485b8ab0c77" instance_seqno: 1786512171153623 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260812 05:22:51.480307 22255 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260812 05:22:51.481285 22396 catalog_manager.cc:1558] T 00000000000000000000000000000000 P bc3fd4361d7f4d16b743b7cef31d5779: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 05:22:51.481333 22396 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 05:22:51.481922 22360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:22:51.474623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46464 (local address 127.31.100.60:35959)
0812 05:22:51.474693 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0812 05:22:51.474695 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 05:22:51.474762 (+ 67us) server_negotiation.cc:407] Connection header received
0812 05:22:51.474853 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:22:51.474854 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:22:51.474863 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:22:51.474890 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:22:51.475575 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:22:51.476275 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:22:51.477331 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:22:51.477639 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:22:51.477678 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:22:51.479192 (+ 1514us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 05:22:51.479194 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 05:22:51.479194 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 05:22:51.479200 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0812 05:22:51.480664 (+ 1464us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:22:51.481082 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:22:51.481083 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:22:51.481084 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:22:51.481139 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:22:51.481338 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:22:51.481339 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:22:51.481339 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:22:51.481511 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 05:22:51.481580 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0812 05:22:51.481775 (+ 195us) server_negotiation.cc:299] Negotiation successful
0812 05:22:51.481846 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260812 05:22:51.482091 22251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:22:51.474544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46182 (local address 127.31.100.61:33635)
0812 05:22:51.474579 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0812 05:22:51.474581 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 05:22:51.474880 (+ 299us) server_negotiation.cc:407] Connection header received
0812 05:22:51.474967 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:22:51.474968 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:22:51.474977 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:22:51.475005 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:22:51.475859 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:22:51.476539 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:22:51.477490 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:22:51.477768 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:22:51.477806 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:22:51.479356 (+ 1550us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 05:22:51.479380 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 05:22:51.479381 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 05:22:51.479388 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 05:22:51.480722 (+ 1334us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:22:51.481126 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:22:51.481128 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:22:51.481128 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:22:51.481188 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:22:51.481377 (+ 189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:22:51.481378 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:22:51.481378 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:22:51.481526 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 05:22:51.481635 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0812 05:22:51.481936 (+ 301us) server_negotiation.cc:299] Negotiation successful
0812 05:22:51.482007 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260812 05:22:51.483615 22136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:22:51.474514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45864 (local address 127.31.100.62:32833)
0812 05:22:51.474553 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0812 05:22:51.474555 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 05:22:51.474787 (+ 232us) server_negotiation.cc:407] Connection header received
0812 05:22:51.474872 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:22:51.474873 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:22:51.474882 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:22:51.474910 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:22:51.475658 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:22:51.476342 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:22:51.477331 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:22:51.477638 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:22:51.477685 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:22:51.479761 (+ 2076us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 05:22:51.479764 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 05:22:51.479764 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 05:22:51.479772 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0812 05:22:51.481699 (+ 1927us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:22:51.482869 (+ 1170us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:22:51.482871 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:22:51.482871 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:22:51.482928 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:22:51.483096 (+ 168us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:22:51.483097 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:22:51.483097 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:22:51.483199 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 05:22:51.483274 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0812 05:22:51.483460 (+ 186us) server_negotiation.cc:299] Negotiation successful
0812 05:22:51.483529 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260812 05:22:51.495677 22128 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.100.60" port: 35959 } attempt: 2
Aug 12 05:22:51 dist-test-slave-42z9 krb5kdc[21990](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512171, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.100.62@KRBTEST.COM for kudu/127.31.100.60@KRBTEST.COM
I20260812 05:22:51.502604 22360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:22:51.496020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46472 (local address 127.31.100.60:35959)
0812 05:22:51.496079 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0812 05:22:51.496081 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 05:22:51.496091 (+ 10us) server_negotiation.cc:407] Connection header received
0812 05:22:51.496129 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:22:51.496130 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:22:51.496140 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:22:51.496193 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:22:51.496453 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:22:51.497192 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:22:51.497928 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:22:51.498242 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:22:51.498310 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:22:51.500227 (+ 1917us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 05:22:51.500228 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 05:22:51.500228 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 05:22:51.500234 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0812 05:22:51.501536 (+ 1302us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:22:51.501881 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:22:51.501881 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:22:51.501882 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:22:51.501936 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:22:51.502141 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:22:51.502142 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:22:51.502142 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:22:51.502257 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 05:22:51.502316 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0812 05:22:51.502482 (+ 166us) server_negotiation.cc:299] Negotiation successful
0812 05:22:51.502543 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260812 05:22:51.502633 22135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:22:51.495961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.60:35959 (local address 127.0.0.1:46472)
0812 05:22:51.496031 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0812 05:22:51.496042 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0812 05:22:51.496086 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:22:51.496199 (+ 113us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:22:51.496200 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:22:51.496328 (+ 128us) client_negotiation.cc:189] Negotiated authn=SASL
0812 05:22:51.496367 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:22:51.496368 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:22:51.497285 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:22:51.497286 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:22:51.497856 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:22:51.497859 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:22:51.498326 (+ 467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:22:51.498327 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:22:51.498375 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:22:51.498797 (+ 422us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 05:22:51.498802 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0812 05:22:51.500123 (+ 1321us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 05:22:51.501605 (+ 1482us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 05:22:51.501606 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 05:22:51.501606 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 05:22:51.501817 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 05:22:51.501989 (+ 172us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 05:22:51.501989 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 05:22:51.501990 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 05:22:51.502058 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 05:22:51.502323 (+ 265us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 05:22:51.502324 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 05:22:51.502369 (+ 45us) client_negotiation.cc:769] Sending connection context
0812 05:22:51.502446 (+ 77us) client_negotiation.cc:240] Negotiation successful
0812 05:22:51.502532 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49}
I20260812 05:22:51.504244 22128 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 06d3c62f634b41a38fadb485b8ab0c77: Bootstrap starting.
I20260812 05:22:51.505017 22128 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 06d3c62f634b41a38fadb485b8ab0c77: Neither blocks nor log segments found. Creating new log.
I20260812 05:22:51.505338 22128 log.cc:824] T 00000000000000000000000000000000 P 06d3c62f634b41a38fadb485b8ab0c77: Log is configured to *not* fsync() on all Append() calls
I20260812 05:22:51.506433 22128 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 06d3c62f634b41a38fadb485b8ab0c77: No bootstrap required, opened a new log
I20260812 05:22:51.508342 22128 raft_consensus.cc:348] T 00000000000000000000000000000000 P 06d3c62f634b41a38fadb485b8ab0c77 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06d3c62f634b41a38fadb485b8ab0c77" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 32833 } } peers { permanent_uuid: "8229f066d5604ad6b94fb1078340f112" member_type: VOTER last_known_addr { host: "127.31.100.61" port: 33635 } } peers { permanent_uuid: "bc3fd4361d7f4d16b743b7cef31d5779" member_type: VOTER last_known_addr { host: "127.31.100.60" port: 35959 } }
I20260812 05:22:51.508478 22128 raft_consensus.cc:374] T 00000000000000000000000000000000 P 06d3c62f634b41a38fadb485b8ab0c77 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 05:22:51.508512 22128 raft_consensus.cc:729] T 00000000000000000000000000000000 P 06d3c62f634b41a38fadb485b8ab0c77 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06d3c62f634b41a38fadb485b8ab0c77, State: Initialized, Role: FOLLOWER
I20260812 05:22:51.508633 22128 consensus_queue.cc:260] T 00000000000000000000000000000000 P 06d3c62f634b41a38fadb485b8ab0c77 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06d3c62f634b41a38fadb485b8ab0c77" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 32833 } } peers { permanent_uuid: "8229f066d5604ad6b94fb1078340f112" member_type: VOTER last_known_addr { host: "127.31.100.61" port: 33635 } } peers { permanent_uuid: "bc3fd4361d7f4d16b743b7cef31d5779" member_type: VOTER last_known_addr { host: "127.31.100.60" port: 35959 } }
I20260812 05:22:51.508910 22403 sys_catalog.cc:455] T 00000000000000000000000000000000 P 06d3c62f634b41a38fadb485b8ab0c77 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06d3c62f634b41a38fadb485b8ab0c77" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 32833 } } peers { permanent_uuid: "8229f066d5604ad6b94fb1078340f112" member_type: VOTER last_known_addr { host: "127.31.100.61" port: 33635 } } peers { permanent_uuid: "bc3fd4361d7f4d16b743b7cef31d5779" member_type: VOTER last_known_addr { host: "127.31.100.60" port: 35959 } } }
I20260812 05:22:51.508949 22128 sys_catalog.cc:564] T 00000000000000000000000000000000 P 06d3c62f634b41a38fadb485b8ab0c77 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 05:22:51.509001 22403 sys_catalog.cc:458] T 00000000000000000000000000000000 P 06d3c62f634b41a38fadb485b8ab0c77 [sys.catalog]: This master's current role is: FOLLOWER
W20260812 05:22:51.510816 22426 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 06d3c62f634b41a38fadb485b8ab0c77: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 05:22:51.510866 22426 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 05:22:51.511682 22009 master_runner.cc:186] Error getting master registration for 127.31.100.61:33635: OK, instance_id { permanent_uuid: "8229f066d5604ad6b94fb1078340f112" instance_seqno: 1786512171294653 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260812 05:22:51.513689 22009 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260812 05:22:51.527372 22247 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.100.60" port: 35959 } attempt: 3
Aug 12 05:22:51 dist-test-slave-42z9 krb5kdc[21990](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512171, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.100.61@KRBTEST.COM for kudu/127.31.100.60@KRBTEST.COM
I20260812 05:22:51.533378 22360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:22:51.527688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46474 (local address 127.31.100.60:35959)
0812 05:22:51.527722 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0812 05:22:51.527724 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 05:22:51.527732 (+ 8us) server_negotiation.cc:407] Connection header received
0812 05:22:51.527793 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:22:51.527794 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:22:51.527803 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:22:51.527849 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:22:51.528138 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:22:51.528601 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:22:51.529251 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:22:51.529464 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:22:51.529524 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:22:51.531197 (+ 1673us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 05:22:51.531198 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 05:22:51.531199 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 05:22:51.531206 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 05:22:51.532405 (+ 1199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:22:51.532712 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:22:51.532714 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:22:51.532714 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:22:51.532764 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:22:51.532930 (+ 166us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:22:51.532931 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:22:51.532931 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:22:51.533031 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 05:22:51.533085 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0812 05:22:51.533225 (+ 140us) server_negotiation.cc:299] Negotiation successful
0812 05:22:51.533288 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260812 05:22:51.533389 22249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:22:51.527639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.60:35959 (local address 127.0.0.1:46474)
0812 05:22:51.527687 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0812 05:22:51.527698 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0812 05:22:51.527738 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:22:51.527852 (+ 114us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:22:51.527853 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:22:51.528023 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0812 05:22:51.528067 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:22:51.528069 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:22:51.528673 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:22:51.528674 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:22:51.529167 (+ 493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:22:51.529168 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:22:51.529528 (+ 360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:22:51.529529 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:22:51.529590 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:22:51.529964 (+ 374us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 05:22:51.529970 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0812 05:22:51.531118 (+ 1148us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 05:22:51.532491 (+ 1373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 05:22:51.532492 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 05:22:51.532492 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 05:22:51.532650 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 05:22:51.532822 (+ 172us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 05:22:51.532822 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 05:22:51.532823 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 05:22:51.532878 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 05:22:51.533089 (+ 211us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 05:22:51.533091 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 05:22:51.533134 (+ 43us) client_negotiation.cc:769] Sending connection context
0812 05:22:51.533193 (+ 59us) client_negotiation.cc:240] Negotiation successful
0812 05:22:51.533287 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260812 05:22:51.534693 22247 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8229f066d5604ad6b94fb1078340f112: Bootstrap starting.
I20260812 05:22:51.535316 22247 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8229f066d5604ad6b94fb1078340f112: Neither blocks nor log segments found. Creating new log.
I20260812 05:22:51.535604 22247 log.cc:824] T 00000000000000000000000000000000 P 8229f066d5604ad6b94fb1078340f112: Log is configured to *not* fsync() on all Append() calls
I20260812 05:22:51.536793 22247 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8229f066d5604ad6b94fb1078340f112: No bootstrap required, opened a new log
I20260812 05:22:51.538216 22247 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8229f066d5604ad6b94fb1078340f112 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06d3c62f634b41a38fadb485b8ab0c77" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 32833 } } peers { permanent_uuid: "8229f066d5604ad6b94fb1078340f112" member_type: VOTER last_known_addr { host: "127.31.100.61" port: 33635 } } peers { permanent_uuid: "bc3fd4361d7f4d16b743b7cef31d5779" member_type: VOTER last_known_addr { host: "127.31.100.60" port: 35959 } }
I20260812 05:22:51.538324 22247 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8229f066d5604ad6b94fb1078340f112 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 05:22:51.538365 22247 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8229f066d5604ad6b94fb1078340f112 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8229f066d5604ad6b94fb1078340f112, State: Initialized, Role: FOLLOWER
I20260812 05:22:51.538486 22247 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8229f066d5604ad6b94fb1078340f112 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06d3c62f634b41a38fadb485b8ab0c77" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 32833 } } peers { permanent_uuid: "8229f066d5604ad6b94fb1078340f112" member_type: VOTER last_known_addr { host: "127.31.100.61" port: 33635 } } peers { permanent_uuid: "bc3fd4361d7f4d16b743b7cef31d5779" member_type: VOTER last_known_addr { host: "127.31.100.60" port: 35959 } }
I20260812 05:22:51.538734 22429 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8229f066d5604ad6b94fb1078340f112 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06d3c62f634b41a38fadb485b8ab0c77" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 32833 } } peers { permanent_uuid: "8229f066d5604ad6b94fb1078340f112" member_type: VOTER last_known_addr { host: "127.31.100.61" port: 33635 } } peers { permanent_uuid: "bc3fd4361d7f4d16b743b7cef31d5779" member_type: VOTER last_known_addr { host: "127.31.100.60" port: 35959 } } }
I20260812 05:22:51.538795 22247 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8229f066d5604ad6b94fb1078340f112 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 05:22:51.538903 22429 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8229f066d5604ad6b94fb1078340f112 [sys.catalog]: This master's current role is: FOLLOWER
W20260812 05:22:51.540515 22449 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8229f066d5604ad6b94fb1078340f112: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 05:22:51.540618 22449 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260812 05:22:51.542608 22174 tablet.cc:2369] T 00000000000000000000000000000000 P 8229f066d5604ad6b94fb1078340f112: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 05:22:51.542896 22144 master_runner.cc:263] No leader master found from master 8229f066d5604ad6b94fb1078340f112
I20260812 05:22:51.542945 22144 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260812 05:22:51.622627 22374 raft_consensus.cc:482] T 00000000000000000000000000000000 P bc3fd4361d7f4d16b743b7cef31d5779 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 05:22:51.622763 22374 raft_consensus.cc:504] T 00000000000000000000000000000000 P bc3fd4361d7f4d16b743b7cef31d5779 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06d3c62f634b41a38fadb485b8ab0c77" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 32833 } } peers { permanent_uuid: "8229f066d5604ad6b94fb1078340f112" member_type: VOTER last_known_addr { host: "127.31.100.61" port: 33635 } } peers { permanent_uuid: "bc3fd4361d7f4d16b743b7cef31d5779" member_type: VOTER last_known_addr { host: "127.31.100.60" port: 35959 } }
I20260812 05:22:51.623027 22374 leader_election.cc:288] T 00000000000000000000000000000000 P bc3fd4361d7f4d16b743b7cef31d5779 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 06d3c62f634b41a38fadb485b8ab0c77 (127.31.100.62:32833), 8229f066d5604ad6b94fb1078340f112 (127.31.100.61:33635)
I20260812 05:22:51.623592 22206 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "bc3fd4361d7f4d16b743b7cef31d5779" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8229f066d5604ad6b94fb1078340f112" is_pre_election: true
I20260812 05:22:51.623775 22206 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 8229f066d5604ad6b94fb1078340f112 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bc3fd4361d7f4d16b743b7cef31d5779 in term 0.
I20260812 05:22:51.623999 22284 leader_election.cc:302] T 00000000000000000000000000000000 P bc3fd4361d7f4d16b743b7cef31d5779 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8229f066d5604ad6b94fb1078340f112, bc3fd4361d7f4d16b743b7cef31d5779; no voters:
I20260812 05:22:51.624262 22068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "bc3fd4361d7f4d16b743b7cef31d5779" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06d3c62f634b41a38fadb485b8ab0c77" is_pre_election: true
I20260812 05:22:51.624415 22068 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 06d3c62f634b41a38fadb485b8ab0c77 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bc3fd4361d7f4d16b743b7cef31d5779 in term 0.
I20260812 05:22:51.624663 22374 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bc3fd4361d7f4d16b743b7cef31d5779 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 05:22:51.624730 22374 raft_consensus.cc:482] T 00000000000000000000000000000000 P bc3fd4361d7f4d16b743b7cef31d5779 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 05:22:51.624760 22374 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bc3fd4361d7f4d16b743b7cef31d5779 [term 0 FOLLOWER]: Advancing to term 1
I20260812 05:22:51.625801 22374 raft_consensus.cc:504] T 00000000000000000000000000000000 P bc3fd4361d7f4d16b743b7cef31d5779 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06d3c62f634b41a38fadb485b8ab0c77" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 32833 } } peers { permanent_uuid: "8229f066d5604ad6b94fb1078340f112" member_type: VOTER last_known_addr { host: "127.31.100.61" port: 33635 } } peers { permanent_uuid: "bc3fd4361d7f4d16b743b7cef31d5779" member_type: VOTER last_known_addr { host: "127.31.100.60" port: 35959 } }
I20260812 05:22:51.625945 22374 leader_election.cc:288] T 00000000000000000000000000000000 P bc3fd4361d7f4d16b743b7cef31d5779 [CANDIDATE]: Term 1 election: Requested vote from peers 06d3c62f634b41a38fadb485b8ab0c77 (127.31.100.62:32833), 8229f066d5604ad6b94fb1078340f112 (127.31.100.61:33635)
I20260812 05:22:51.626313 22206 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "bc3fd4361d7f4d16b743b7cef31d5779" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8229f066d5604ad6b94fb1078340f112"
I20260812 05:22:51.626401 22206 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8229f066d5604ad6b94fb1078340f112 [term 0 FOLLOWER]: Advancing to term 1
I20260812 05:22:51.627293 22206 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 8229f066d5604ad6b94fb1078340f112 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bc3fd4361d7f4d16b743b7cef31d5779 in term 1.
I20260812 05:22:51.627462 22284 leader_election.cc:302] T 00000000000000000000000000000000 P bc3fd4361d7f4d16b743b7cef31d5779 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8229f066d5604ad6b94fb1078340f112, bc3fd4361d7f4d16b743b7cef31d5779; no voters:
I20260812 05:22:51.627595 22068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "bc3fd4361d7f4d16b743b7cef31d5779" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06d3c62f634b41a38fadb485b8ab0c77"
I20260812 05:22:51.627687 22068 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 06d3c62f634b41a38fadb485b8ab0c77 [term 0 FOLLOWER]: Advancing to term 1
I20260812 05:22:51.628595 22068 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 06d3c62f634b41a38fadb485b8ab0c77 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bc3fd4361d7f4d16b743b7cef31d5779 in term 1.
I20260812 05:22:51.629745 22374 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bc3fd4361d7f4d16b743b7cef31d5779 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 05:22:51.629909 22374 raft_consensus.cc:686] T 00000000000000000000000000000000 P bc3fd4361d7f4d16b743b7cef31d5779 [term 1 LEADER]: Becoming Leader. State: Replica: bc3fd4361d7f4d16b743b7cef31d5779, State: Running, Role: LEADER
I20260812 05:22:51.630008 22374 consensus_queue.cc:237] T 00000000000000000000000000000000 P bc3fd4361d7f4d16b743b7cef31d5779 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06d3c62f634b41a38fadb485b8ab0c77" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 32833 } } peers { permanent_uuid: "8229f066d5604ad6b94fb1078340f112" member_type: VOTER last_known_addr { host: "127.31.100.61" port: 33635 } } peers { permanent_uuid: "bc3fd4361d7f4d16b743b7cef31d5779" member_type: VOTER last_known_addr { host: "127.31.100.60" port: 35959 } }
I20260812 05:22:51.630396 22374 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc3fd4361d7f4d16b743b7cef31d5779 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bc3fd4361d7f4d16b743b7cef31d5779. Latest consensus state: current_term: 1 leader_uuid: "bc3fd4361d7f4d16b743b7cef31d5779" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06d3c62f634b41a38fadb485b8ab0c77" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 32833 } } peers { permanent_uuid: "8229f066d5604ad6b94fb1078340f112" member_type: VOTER last_known_addr { host: "127.31.100.61" port: 33635 } } peers { permanent_uuid: "bc3fd4361d7f4d16b743b7cef31d5779" member_type: VOTER last_known_addr { host: "127.31.100.60" port: 35959 } } }
I20260812 05:22:51.630474 22374 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc3fd4361d7f4d16b743b7cef31d5779 [sys.catalog]: This master's current role is: LEADER
I20260812 05:22:51.630708 22457 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 05:22:51.630908 22457 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 05:22:51.632838 22206 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 8229f066d5604ad6b94fb1078340f112 [term 1 FOLLOWER]: Refusing update from remote peer bc3fd4361d7f4d16b743b7cef31d5779: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 05:22:51.633145 22374 consensus_queue.cc:1035] T 00000000000000000000000000000000 P bc3fd4361d7f4d16b743b7cef31d5779 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8229f066d5604ad6b94fb1078340f112" member_type: VOTER last_known_addr { host: "127.31.100.61" port: 33635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 05:22:51.634423 22429 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8229f066d5604ad6b94fb1078340f112 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bc3fd4361d7f4d16b743b7cef31d5779. Latest consensus state: current_term: 1 leader_uuid: "bc3fd4361d7f4d16b743b7cef31d5779" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06d3c62f634b41a38fadb485b8ab0c77" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 32833 } } peers { permanent_uuid: "8229f066d5604ad6b94fb1078340f112" member_type: VOTER last_known_addr { host: "127.31.100.61" port: 33635 } } peers { permanent_uuid: "bc3fd4361d7f4d16b743b7cef31d5779" member_type: VOTER last_known_addr { host: "127.31.100.60" port: 35959 } } }
I20260812 05:22:51.634519 22429 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8229f066d5604ad6b94fb1078340f112 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 05:22:51.634886 22068 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 06d3c62f634b41a38fadb485b8ab0c77 [term 1 FOLLOWER]: Refusing update from remote peer bc3fd4361d7f4d16b743b7cef31d5779: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 05:22:51.635154 22374 consensus_queue.cc:1035] T 00000000000000000000000000000000 P bc3fd4361d7f4d16b743b7cef31d5779 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06d3c62f634b41a38fadb485b8ab0c77" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 32833 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 05:22:51.636312 22403 sys_catalog.cc:455] T 00000000000000000000000000000000 P 06d3c62f634b41a38fadb485b8ab0c77 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bc3fd4361d7f4d16b743b7cef31d5779. Latest consensus state: current_term: 1 leader_uuid: "bc3fd4361d7f4d16b743b7cef31d5779" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06d3c62f634b41a38fadb485b8ab0c77" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 32833 } } peers { permanent_uuid: "8229f066d5604ad6b94fb1078340f112" member_type: VOTER last_known_addr { host: "127.31.100.61" port: 33635 } } peers { permanent_uuid: "bc3fd4361d7f4d16b743b7cef31d5779" member_type: VOTER last_known_addr { host: "127.31.100.60" port: 35959 } } }
I20260812 05:22:51.636438 22403 sys_catalog.cc:458] T 00000000000000000000000000000000 P 06d3c62f634b41a38fadb485b8ab0c77 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 05:22:51.637405 22458 mvcc.cc:205] Tried to move back new op lower bound from 7317553855005929472 to 7317553854997123072. Current Snapshot: MvccSnapshot[applied={T|T < 7317553855005929472}]
I20260812 05:22:51.637678 22374 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc3fd4361d7f4d16b743b7cef31d5779 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "bc3fd4361d7f4d16b743b7cef31d5779" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06d3c62f634b41a38fadb485b8ab0c77" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 32833 } } peers { permanent_uuid: "8229f066d5604ad6b94fb1078340f112" member_type: VOTER last_known_addr { host: "127.31.100.61" port: 33635 } } peers { permanent_uuid: "bc3fd4361d7f4d16b743b7cef31d5779" member_type: VOTER last_known_addr { host: "127.31.100.60" port: 35959 } } }
I20260812 05:22:51.637763 22374 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc3fd4361d7f4d16b743b7cef31d5779 [sys.catalog]: This master's current role is: LEADER
I20260812 05:22:51.638087 22403 sys_catalog.cc:455] T 00000000000000000000000000000000 P 06d3c62f634b41a38fadb485b8ab0c77 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "bc3fd4361d7f4d16b743b7cef31d5779" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06d3c62f634b41a38fadb485b8ab0c77" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 32833 } } peers { permanent_uuid: "8229f066d5604ad6b94fb1078340f112" member_type: VOTER last_known_addr { host: "127.31.100.61" port: 33635 } } peers { permanent_uuid: "bc3fd4361d7f4d16b743b7cef31d5779" member_type: VOTER last_known_addr { host: "127.31.100.60" port: 35959 } } }
I20260812 05:22:51.638180 22403 sys_catalog.cc:458] T 00000000000000000000000000000000 P 06d3c62f634b41a38fadb485b8ab0c77 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 05:22:51.638803 22463 mvcc.cc:205] Tried to move back new op lower bound from 7317553855005929472 to 7317553854997123072. Current Snapshot: MvccSnapshot[applied={T|T < 7317553855005929472}]
I20260812 05:22:51.639626 22374 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc3fd4361d7f4d16b743b7cef31d5779 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "bc3fd4361d7f4d16b743b7cef31d5779" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06d3c62f634b41a38fadb485b8ab0c77" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 32833 } } peers { permanent_uuid: "8229f066d5604ad6b94fb1078340f112" member_type: VOTER last_known_addr { host: "127.31.100.61" port: 33635 } } peers { permanent_uuid: "bc3fd4361d7f4d16b743b7cef31d5779" member_type: VOTER last_known_addr { host: "127.31.100.60" port: 35959 } } }
I20260812 05:22:51.639705 22374 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc3fd4361d7f4d16b743b7cef31d5779 [sys.catalog]: This master's current role is: LEADER
I20260812 05:22:51.640025 22429 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8229f066d5604ad6b94fb1078340f112 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "bc3fd4361d7f4d16b743b7cef31d5779" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06d3c62f634b41a38fadb485b8ab0c77" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 32833 } } peers { permanent_uuid: "8229f066d5604ad6b94fb1078340f112" member_type: VOTER last_known_addr { host: "127.31.100.61" port: 33635 } } peers { permanent_uuid: "bc3fd4361d7f4d16b743b7cef31d5779" member_type: VOTER last_known_addr { host: "127.31.100.60" port: 35959 } } }
I20260812 05:22:51.640117 22429 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8229f066d5604ad6b94fb1078340f112 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 05:22:51.640745 22460 mvcc.cc:205] Tried to move back new op lower bound from 7317553855005929472 to 7317553854997123072. Current Snapshot: MvccSnapshot[applied={T|T < 7317553855005929472}]
I20260812 05:22:51.641845 22457 catalog_manager.cc:1347] Generated new cluster ID: a4e91d44a0854f4ca440d744cabae327
I20260812 05:22:51.641906 22457 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 05:22:51.662631 22457 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 05:22:51.663415 22457 catalog_manager.cc:1504] Loading token signing keys...
I20260812 05:22:51.671034 22457 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bc3fd4361d7f4d16b743b7cef31d5779: Generated new TSK 0
I20260812 05:22:51.671226 22457 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 05:22:52.511283 22426 catalog_manager.cc:1259] Loaded cluster ID: a4e91d44a0854f4ca440d744cabae327
I20260812 05:22:52.511359 22426 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 06d3c62f634b41a38fadb485b8ab0c77: loading cluster ID for follower catalog manager: success
I20260812 05:22:52.512135 22426 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 06d3c62f634b41a38fadb485b8ab0c77: acquiring CA information for follower catalog manager: success
I20260812 05:22:52.512387 22426 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 06d3c62f634b41a38fadb485b8ab0c77: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260812 05:22:52.541098 22449 catalog_manager.cc:1259] Loaded cluster ID: a4e91d44a0854f4ca440d744cabae327
I20260812 05:22:52.541172 22449 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 8229f066d5604ad6b94fb1078340f112: loading cluster ID for follower catalog manager: success
I20260812 05:22:52.541960 22449 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 8229f066d5604ad6b94fb1078340f112: acquiring CA information for follower catalog manager: success
I20260812 05:22:52.542222 22449 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 8229f066d5604ad6b94fb1078340f112: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260812 05:22:52.732770 32144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0waFvT/build/release/bin/kudu with pid 22009
I20260812 05:22:52.747292 32144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0waFvT/build/release/bin/kudu with pid 22144
I20260812 05:22:52.754575 32144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0waFvT/build/release/bin/kudu with pid 22255
2026-08-12T05:22:52Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (3024 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786512017613366-32144-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 05:22:52 dist-test-slave-42z9 krb5kdc[22481](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 05:22:52 dist-test-slave-42z9 krb5kdc[22481](info): set up 2 sockets
Aug 12 05:22:52 dist-test-slave-42z9 krb5kdc[22481](info): commencing operation
krb5kdc: starting...
W20260812 05:22:53.910393 32144 mini_kdc.cc:121] Time spent starting KDC: real 1.133s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786512017613366-32144-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786512017613366-32144-0/krb5kdc/test-user.keytab.
Aug 12 05:22:53 dist-test-slave-42z9 krb5kdc[22481](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512173, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T05:22:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T05:22:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.100.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.100.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.100.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.62.keytab.
Entry for principal kudu/127.31.100.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.62.keytab.
I20260812 05:22:54.021281 32144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0waFvT/build/release/bin/kudu
/tmp/dist-test-task0waFvT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786512017613366-32144-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786512017613366-32144-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786512017613366-32144-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786512017613366-32144-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.100.62:34357
--webserver_interface=127.31.100.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.62.keytab
--principal=kudu/127.31.100.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.100.20:46359
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.100.62:34357
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786512017613366-32144-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786512017613366-32144-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786512017613366-32144-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 05:22:54.101475 22499 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 05:22:54.101670 22499 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 05:22:54.101698 22499 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 05:22:54.102836 22499 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 05:22:54.102877 22499 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 05:22:54.102895 22499 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 05:22:54.102910 22499 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 05:22:54.102926 22499 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 05:22:54.104413 22499 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.100.20:46359
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786512017613366-32144-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786512017613366-32144-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.100.62:34357
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.62.keytab
--principal=kudu/127.31.100.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.100.62:34357
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786512017613366-32144-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.100.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22499
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786512017613366-32144-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 04:20:34 UTC on ccycloud.quasar-phqdqp.root.comops.site
build id 4581871
I20260812 05:22:54.104660 22499 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 05:22:54.104868 22499 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 05:22:54.109540 22515 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 05:22:54.109524 22517 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 05:22:54.109694 22499 server_base.cc:1034] running on GCE node
W20260812 05:22:54.109570 22513 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 05:22:54.110003 22499 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 05:22:54.110280 22499 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 05:22:54.111483 22499 hybrid_clock.cc:648] HybridClock initialized: now 1786512174111418 us; error 34 us; skew 500 ppm
Aug 12 05:22:54 dist-test-slave-42z9 krb5kdc[22481](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512174, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.100.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 05:22:54.113915 22499 init.cc:377] Logged in from keytab as kudu/127.31.100.62@KRBTEST.COM (short username kudu)
I20260812 05:22:54.114358 22499 webserver.cc:460] Webserver started at http://127.31.100.62:46211/ using document root <none> and password file <none>
I20260812 05:22:54.114606 22499 fs_manager.cc:359] Metadata directory not provided
I20260812 05:22:54.114656 22499 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 05:22:54.114784 22499 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 05:22:54.115767 22499 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786512017613366-32144-0/minicluster-data/master-0/data/instance:
uuid: "c4d8f52f58814e4db0d366082ef7a839"
format_stamp: "Formatted at 2026-08-12 05:22:54 on dist-test-slave-42z9"
server_key: "b25635d8ae37729e8fe0b959b20042af"
server_key_iv: "27052b200466dc04c30e44d0bdb1245b"
server_key_version: "encryptionkey@0"
I20260812 05:22:54.116080 22499 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786512017613366-32144-0/minicluster-data/master-0/wal/instance:
uuid: "c4d8f52f58814e4db0d366082ef7a839"
format_stamp: "Formatted at 2026-08-12 05:22:54 on dist-test-slave-42z9"
server_key: "b25635d8ae37729e8fe0b959b20042af"
server_key_iv: "27052b200466dc04c30e44d0bdb1245b"
server_key_version: "encryptionkey@0"
I20260812 05:22:54.117260 22499 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260812 05:22:54.117944 22531 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 05:22:54.118094 22499 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260812 05:22:54.118180 22499 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786512017613366-32144-0/minicluster-data/master-0/data,/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786512017613366-32144-0/minicluster-data/master-0/wal
uuid: "c4d8f52f58814e4db0d366082ef7a839"
format_stamp: "Formatted at 2026-08-12 05:22:54 on dist-test-slave-42z9"
server_key: "b25635d8ae37729e8fe0b959b20042af"
server_key_iv: "27052b200466dc04c30e44d0bdb1245b"
server_key_version: "encryptionkey@0"
I20260812 05:22:54.118242 22499 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786512017613366-32144-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786512017613366-32144-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786512017613366-32144-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 05:22:54.128710 22499 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 05:22:54.130586 22499 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 05:22:54.130712 22499 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 05:22:54.134552 22499 rpc_server.cc:307] RPC server started. Bound to: 127.31.100.62:34357
I20260812 05:22:54.134593 22633 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.100.62:34357 every 8 connection(s)
I20260812 05:22:54.134912 22499 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786512017613366-32144-0/minicluster-data/master-0/data/info.pb
I20260812 05:22:54.135633 32144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0waFvT/build/release/bin/kudu as pid 22499
I20260812 05:22:54.135712 32144 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786512017613366-32144-0/minicluster-data/master-0/wal/instance
I20260812 05:22:54.135792 22636 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 05:22:54.135892 32144 external_mini_cluster.cc:1428] Setting key 987c1ff2841d58b4a5ca9373982a6885
I20260812 05:22:54.138957 22636 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c4d8f52f58814e4db0d366082ef7a839: Bootstrap starting.
I20260812 05:22:54.139637 22636 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c4d8f52f58814e4db0d366082ef7a839: Neither blocks nor log segments found. Creating new log.
I20260812 05:22:54.139905 22636 log.cc:824] T 00000000000000000000000000000000 P c4d8f52f58814e4db0d366082ef7a839: Log is configured to *not* fsync() on all Append() calls
I20260812 05:22:54.140560 22636 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c4d8f52f58814e4db0d366082ef7a839: No bootstrap required, opened a new log
Aug 12 05:22:54 dist-test-slave-42z9 krb5kdc[22481](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512173, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.100.62@KRBTEST.COM
I20260812 05:22:54.142736 22636 raft_consensus.cc:348] T 00000000000000000000000000000000 P c4d8f52f58814e4db0d366082ef7a839 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4d8f52f58814e4db0d366082ef7a839" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 34357 } }
I20260812 05:22:54.142902 22636 raft_consensus.cc:374] T 00000000000000000000000000000000 P c4d8f52f58814e4db0d366082ef7a839 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 05:22:54.142958 22636 raft_consensus.cc:729] T 00000000000000000000000000000000 P c4d8f52f58814e4db0d366082ef7a839 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4d8f52f58814e4db0d366082ef7a839, State: Initialized, Role: FOLLOWER
I20260812 05:22:54.143091 22636 consensus_queue.cc:260] T 00000000000000000000000000000000 P c4d8f52f58814e4db0d366082ef7a839 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4d8f52f58814e4db0d366082ef7a839" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 34357 } }
I20260812 05:22:54.143180 22636 raft_consensus.cc:388] T 00000000000000000000000000000000 P c4d8f52f58814e4db0d366082ef7a839 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 05:22:54.143224 22636 raft_consensus.cc:482] T 00000000000000000000000000000000 P c4d8f52f58814e4db0d366082ef7a839 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 05:22:54.143271 22636 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c4d8f52f58814e4db0d366082ef7a839 [term 0 FOLLOWER]: Advancing to term 1
I20260812 05:22:54.143877 22636 raft_consensus.cc:504] T 00000000000000000000000000000000 P c4d8f52f58814e4db0d366082ef7a839 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4d8f52f58814e4db0d366082ef7a839" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 34357 } }
I20260812 05:22:54.143990 22636 leader_election.cc:302] T 00000000000000000000000000000000 P c4d8f52f58814e4db0d366082ef7a839 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c4d8f52f58814e4db0d366082ef7a839; no voters:
I20260812 05:22:54.144160 22636 leader_election.cc:288] T 00000000000000000000000000000000 P c4d8f52f58814e4db0d366082ef7a839 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 05:22:54.144207 22643 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c4d8f52f58814e4db0d366082ef7a839 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 05:22:54.144394 22643 raft_consensus.cc:686] T 00000000000000000000000000000000 P c4d8f52f58814e4db0d366082ef7a839 [term 1 LEADER]: Becoming Leader. State: Replica: c4d8f52f58814e4db0d366082ef7a839, State: Running, Role: LEADER
I20260812 05:22:54.144441 22636 sys_catalog.cc:564] T 00000000000000000000000000000000 P c4d8f52f58814e4db0d366082ef7a839 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 05:22:54.144560 22643 consensus_queue.cc:237] T 00000000000000000000000000000000 P c4d8f52f58814e4db0d366082ef7a839 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4d8f52f58814e4db0d366082ef7a839" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 34357 } }
I20260812 05:22:54.144936 22645 sys_catalog.cc:455] T 00000000000000000000000000000000 P c4d8f52f58814e4db0d366082ef7a839 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c4d8f52f58814e4db0d366082ef7a839" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4d8f52f58814e4db0d366082ef7a839" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 34357 } } }
I20260812 05:22:54.145015 22645 sys_catalog.cc:458] T 00000000000000000000000000000000 P c4d8f52f58814e4db0d366082ef7a839 [sys.catalog]: This master's current role is: LEADER
I20260812 05:22:54.145166 22648 sys_catalog.cc:455] T 00000000000000000000000000000000 P c4d8f52f58814e4db0d366082ef7a839 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c4d8f52f58814e4db0d366082ef7a839. Latest consensus state: current_term: 1 leader_uuid: "c4d8f52f58814e4db0d366082ef7a839" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4d8f52f58814e4db0d366082ef7a839" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 34357 } } }
I20260812 05:22:54.145239 22648 sys_catalog.cc:458] T 00000000000000000000000000000000 P c4d8f52f58814e4db0d366082ef7a839 [sys.catalog]: This master's current role is: LEADER
I20260812 05:22:54.145339 22661 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 05:22:54.146965 22661 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 05:22:54.147253 22639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:22:54.136340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56136 (local address 127.31.100.62:34357)
0812 05:22:54.136541 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0812 05:22:54.136550 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 05:22:54.136582 (+ 32us) server_negotiation.cc:407] Connection header received
0812 05:22:54.137226 (+ 644us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:22:54.137235 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:22:54.137297 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:22:54.137457 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:22:54.138055 (+ 598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:22:54.139402 (+ 1347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:22:54.140154 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:22:54.140624 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:22:54.140738 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:22:54.143200 (+ 2462us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 05:22:54.143204 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 05:22:54.143204 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 05:22:54.143217 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0812 05:22:54.145064 (+ 1847us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:22:54.145512 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:22:54.145516 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:22:54.145517 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:22:54.145633 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:22:54.145871 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:22:54.145873 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:22:54.145873 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:22:54.146208 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 05:22:54.146859 (+ 651us) server_negotiation.cc:1035] Waiting for connection context
0812 05:22:54.146933 (+ 74us) server_negotiation.cc:299] Negotiation successful
0812 05:22:54.147066 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":92,"threads_started":1}
I20260812 05:22:54.148152 32144 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 05:22:54.148442 22661 catalog_manager.cc:1347] Generated new cluster ID: 314ced2cffbf478885b6cfd485c8cf77
I20260812 05:22:54.148487 22661 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 05:22:54.152927 22661 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 05:22:54.153406 22661 catalog_manager.cc:1504] Loading token signing keys...
I20260812 05:22:54.156229 22661 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c4d8f52f58814e4db0d366082ef7a839: Generated new TSK 0
I20260812 05:22:54.156369 22661 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 05:22:54.201895 32144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0waFvT/build/release/bin/kudu with pid 22499
2026-08-12T05:22:54Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1450 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786512017613366-32144-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 05:22:54 dist-test-slave-42z9 krb5kdc[22682](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 05:22:54 dist-test-slave-42z9 krb5kdc[22682](info): set up 2 sockets
Aug 12 05:22:54 dist-test-slave-42z9 krb5kdc[22682](info): commencing operation
krb5kdc: starting...
W20260812 05:22:55.373656 32144 mini_kdc.cc:121] Time spent starting KDC: real 1.146s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786512017613366-32144-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786512017613366-32144-0/krb5kdc/test-user.keytab.
Aug 12 05:22:55 dist-test-slave-42z9 krb5kdc[22682](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512175, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T05:22:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T05:22:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.100.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.100.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.100.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.62.keytab.
Entry for principal kudu/127.31.100.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.62.keytab.
I20260812 05:22:55.488015 32144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0waFvT/build/release/bin/kudu
/tmp/dist-test-task0waFvT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786512017613366-32144-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786512017613366-32144-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786512017613366-32144-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786512017613366-32144-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.100.62:40577
--webserver_interface=127.31.100.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.62.keytab
--principal=kudu/127.31.100.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.100.20:37413
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.100.62:40577
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786512017613366-32144-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786512017613366-32144-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786512017613366-32144-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 05:22:55.567718 22705 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 05:22:55.567917 22705 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 05:22:55.567946 22705 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 05:22:55.569067 22705 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 05:22:55.569105 22705 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 05:22:55.569123 22705 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 05:22:55.569140 22705 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 05:22:55.569154 22705 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 05:22:55.570614 22705 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.100.20:37413
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786512017613366-32144-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786512017613366-32144-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.100.62:40577
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.62.keytab
--principal=kudu/127.31.100.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.100.62:40577
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786512017613366-32144-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.100.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22705
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786512017613366-32144-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 04:20:34 UTC on ccycloud.quasar-phqdqp.root.comops.site
build id 4581871
I20260812 05:22:55.570849 22705 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 05:22:55.571054 22705 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 05:22:55.575702 22713 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 05:22:55.575814 22705 server_base.cc:1034] running on GCE node
W20260812 05:22:55.575680 22718 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 05:22:55.575697 22716 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 05:22:55.576113 22705 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 05:22:55.576349 22705 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 05:22:55.577489 22705 hybrid_clock.cc:648] HybridClock initialized: now 1786512175577475 us; error 29 us; skew 500 ppm
Aug 12 05:22:55 dist-test-slave-42z9 krb5kdc[22682](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512175, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.100.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 05:22:55.579779 22705 init.cc:377] Logged in from keytab as kudu/127.31.100.62@KRBTEST.COM (short username kudu)
I20260812 05:22:55.580255 22705 webserver.cc:460] Webserver started at http://127.31.100.62:43853/ using document root <none> and password file <none>
I20260812 05:22:55.580425 22705 fs_manager.cc:359] Metadata directory not provided
I20260812 05:22:55.580463 22705 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 05:22:55.580549 22705 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 05:22:55.581535 22705 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786512017613366-32144-0/minicluster-data/master-0/data/instance:
uuid: "c866d37f66fb4b64bae37566f2baad8e"
format_stamp: "Formatted at 2026-08-12 05:22:55 on dist-test-slave-42z9"
server_key: "9d146bbc124be7926356b761a88170c3"
server_key_iv: "59d13cc9c7748afda4d9392b9aad4348"
server_key_version: "encryptionkey@0"
I20260812 05:22:55.581831 22705 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786512017613366-32144-0/minicluster-data/master-0/wal/instance:
uuid: "c866d37f66fb4b64bae37566f2baad8e"
format_stamp: "Formatted at 2026-08-12 05:22:55 on dist-test-slave-42z9"
server_key: "9d146bbc124be7926356b761a88170c3"
server_key_iv: "59d13cc9c7748afda4d9392b9aad4348"
server_key_version: "encryptionkey@0"
I20260812 05:22:55.582971 22705 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260812 05:22:55.583619 22733 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 05:22:55.583760 22705 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260812 05:22:55.583839 22705 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786512017613366-32144-0/minicluster-data/master-0/data,/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786512017613366-32144-0/minicluster-data/master-0/wal
uuid: "c866d37f66fb4b64bae37566f2baad8e"
format_stamp: "Formatted at 2026-08-12 05:22:55 on dist-test-slave-42z9"
server_key: "9d146bbc124be7926356b761a88170c3"
server_key_iv: "59d13cc9c7748afda4d9392b9aad4348"
server_key_version: "encryptionkey@0"
I20260812 05:22:55.583899 22705 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786512017613366-32144-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786512017613366-32144-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786512017613366-32144-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 05:22:55.602521 22705 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 05:22:55.604128 22705 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 05:22:55.604246 22705 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 05:22:55.608112 22705 rpc_server.cc:307] RPC server started. Bound to: 127.31.100.62:40577
I20260812 05:22:55.608145 22829 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.100.62:40577 every 8 connection(s)
I20260812 05:22:55.608443 22705 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786512017613366-32144-0/minicluster-data/master-0/data/info.pb
I20260812 05:22:55.609361 22830 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 05:22:55.611616 22830 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c866d37f66fb4b64bae37566f2baad8e: Bootstrap starting.
I20260812 05:22:55.612138 22830 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c866d37f66fb4b64bae37566f2baad8e: Neither blocks nor log segments found. Creating new log.
I20260812 05:22:55.612371 22830 log.cc:824] T 00000000000000000000000000000000 P c866d37f66fb4b64bae37566f2baad8e: Log is configured to *not* fsync() on all Append() calls
I20260812 05:22:55.612541 32144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0waFvT/build/release/bin/kudu as pid 22705
I20260812 05:22:55.612651 32144 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786512017613366-32144-0/minicluster-data/master-0/wal/instance
I20260812 05:22:55.612849 32144 external_mini_cluster.cc:1428] Setting key b73e41963861cdb8497c9d4b82ab5ae9
I20260812 05:22:55.613106 22830 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c866d37f66fb4b64bae37566f2baad8e: No bootstrap required, opened a new log
I20260812 05:22:55.615152 22830 raft_consensus.cc:348] T 00000000000000000000000000000000 P c866d37f66fb4b64bae37566f2baad8e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c866d37f66fb4b64bae37566f2baad8e" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 40577 } }
I20260812 05:22:55.615324 22830 raft_consensus.cc:374] T 00000000000000000000000000000000 P c866d37f66fb4b64bae37566f2baad8e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 05:22:55.615370 22830 raft_consensus.cc:729] T 00000000000000000000000000000000 P c866d37f66fb4b64bae37566f2baad8e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c866d37f66fb4b64bae37566f2baad8e, State: Initialized, Role: FOLLOWER
I20260812 05:22:55.615514 22830 consensus_queue.cc:260] T 00000000000000000000000000000000 P c866d37f66fb4b64bae37566f2baad8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c866d37f66fb4b64bae37566f2baad8e" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 40577 } }
I20260812 05:22:55.615592 22830 raft_consensus.cc:388] T 00000000000000000000000000000000 P c866d37f66fb4b64bae37566f2baad8e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 05:22:55.615628 22830 raft_consensus.cc:482] T 00000000000000000000000000000000 P c866d37f66fb4b64bae37566f2baad8e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 05:22:55.615670 22830 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c866d37f66fb4b64bae37566f2baad8e [term 0 FOLLOWER]: Advancing to term 1
I20260812 05:22:55.616309 22830 raft_consensus.cc:504] T 00000000000000000000000000000000 P c866d37f66fb4b64bae37566f2baad8e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c866d37f66fb4b64bae37566f2baad8e" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 40577 } }
I20260812 05:22:55.616413 22830 leader_election.cc:302] T 00000000000000000000000000000000 P c866d37f66fb4b64bae37566f2baad8e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c866d37f66fb4b64bae37566f2baad8e; no voters:
I20260812 05:22:55.616556 22830 leader_election.cc:288] T 00000000000000000000000000000000 P c866d37f66fb4b64bae37566f2baad8e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 05:22:55.616590 22838 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c866d37f66fb4b64bae37566f2baad8e [term 1 FOLLOWER]: Leader election won for term 1
I20260812 05:22:55.616781 22830 sys_catalog.cc:564] T 00000000000000000000000000000000 P c866d37f66fb4b64bae37566f2baad8e [sys.catalog]: configured and running, proceeding with master startup.
I20260812 05:22:55.616873 22838 raft_consensus.cc:686] T 00000000000000000000000000000000 P c866d37f66fb4b64bae37566f2baad8e [term 1 LEADER]: Becoming Leader. State: Replica: c866d37f66fb4b64bae37566f2baad8e, State: Running, Role: LEADER
I20260812 05:22:55.616986 22838 consensus_queue.cc:237] T 00000000000000000000000000000000 P c866d37f66fb4b64bae37566f2baad8e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c866d37f66fb4b64bae37566f2baad8e" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 40577 } }
I20260812 05:22:55.617307 22838 sys_catalog.cc:455] T 00000000000000000000000000000000 P c866d37f66fb4b64bae37566f2baad8e [sys.catalog]: SysCatalogTable state changed. Reason: New leader c866d37f66fb4b64bae37566f2baad8e. Latest consensus state: current_term: 1 leader_uuid: "c866d37f66fb4b64bae37566f2baad8e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c866d37f66fb4b64bae37566f2baad8e" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 40577 } } }
I20260812 05:22:55.617391 22841 sys_catalog.cc:455] T 00000000000000000000000000000000 P c866d37f66fb4b64bae37566f2baad8e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c866d37f66fb4b64bae37566f2baad8e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c866d37f66fb4b64bae37566f2baad8e" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 40577 } } }
I20260812 05:22:55.617467 22841 sys_catalog.cc:458] T 00000000000000000000000000000000 P c866d37f66fb4b64bae37566f2baad8e [sys.catalog]: This master's current role is: LEADER
I20260812 05:22:55.617422 22838 sys_catalog.cc:458] T 00000000000000000000000000000000 P c866d37f66fb4b64bae37566f2baad8e [sys.catalog]: This master's current role is: LEADER
I20260812 05:22:55.617681 22853 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 05:22:55.618261 22853 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 05:22:55.619882 22853 catalog_manager.cc:1347] Generated new cluster ID: ece460ce7ad94ebfacddc2bb26fd457b
I20260812 05:22:55.619932 22853 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 12 05:22:55 dist-test-slave-42z9 krb5kdc[22682](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512175, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.100.62@KRBTEST.COM
I20260812 05:22:55.624361 22837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:22:55.613326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59828 (local address 127.31.100.62:40577)
0812 05:22:55.613552 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0812 05:22:55.613562 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 05:22:55.613595 (+ 33us) server_negotiation.cc:407] Connection header received
0812 05:22:55.614378 (+ 783us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:22:55.614387 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:22:55.614447 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:22:55.614583 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:22:55.615115 (+ 532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:22:55.616702 (+ 1587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:22:55.617759 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:22:55.618287 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:22:55.618382 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:22:55.621123 (+ 2741us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 05:22:55.621127 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 05:22:55.621127 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 05:22:55.621137 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0812 05:22:55.622890 (+ 1753us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:22:55.623263 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:22:55.623265 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:22:55.623266 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:22:55.623356 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:22:55.623552 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:22:55.623552 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:22:55.623553 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:22:55.623809 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 05:22:55.623952 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0812 05:22:55.624058 (+ 106us) server_negotiation.cc:299] Negotiation successful
0812 05:22:55.624180 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":175,"thread_start_us":128,"threads_started":1}
I20260812 05:22:55.627938 22853 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 05:22:55.628438 22853 catalog_manager.cc:1504] Loading token signing keys...
I20260812 05:22:55.632001 22853 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c866d37f66fb4b64bae37566f2baad8e: Generated new TSK 0
I20260812 05:22:55.632148 22853 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.31.100.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.100.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.100.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.1.keytab.
Entry for principal kudu/127.31.100.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.1.keytab.
I20260812 05:22:55.658370 32144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0waFvT/build/release/bin/kudu
/tmp/dist-test-task0waFvT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786512017613366-32144-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786512017613366-32144-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786512017613366-32144-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786512017613366-32144-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.100.1:0
--local_ip_for_outbound_sockets=127.31.100.1
--webserver_interface=127.31.100.1
--webserver_port=0
--tserver_master_addrs=127.31.100.62:40577
--keytab_file=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.1.keytab
--principal=kudu/127.31.100.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.100.20:37413
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786512017613366-32144-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786512017613366-32144-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786512017613366-32144-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 05:22:55.737920 22876 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 05:22:55.738113 22876 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 05:22:55.738145 22876 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 05:22:55.739248 22876 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 05:22:55.739287 22876 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 05:22:55.739322 22876 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.100.1
I20260812 05:22:55.740734 22876 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.100.20:37413
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786512017613366-32144-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786512017613366-32144-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.1.keytab
--principal=kudu/127.31.100.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.100.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786512017613366-32144-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.100.1
--webserver_port=0
--tserver_master_addrs=127.31.100.62:40577
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22876
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.100.1
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786512017613366-32144-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 04:20:34 UTC on ccycloud.quasar-phqdqp.root.comops.site
build id 4581871
I20260812 05:22:55.740962 22876 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 05:22:55.741178 22876 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 05:22:55.745777 22891 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 05:22:55.745772 22886 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 05:22:55.745841 22888 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 05:22:55.746083 22876 server_base.cc:1034] running on GCE node
I20260812 05:22:55.746290 22876 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 05:22:55.746502 22876 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 05:22:55.747634 22876 hybrid_clock.cc:648] HybridClock initialized: now 1786512175747624 us; error 26 us; skew 500 ppm
Aug 12 05:22:55 dist-test-slave-42z9 krb5kdc[22682](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512175, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.100.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 05:22:55.749998 22876 init.cc:377] Logged in from keytab as kudu/127.31.100.1@KRBTEST.COM (short username kudu)
I20260812 05:22:55.750376 22876 webserver.cc:460] Webserver started at http://127.31.100.1:43341/ using document root <none> and password file <none>
I20260812 05:22:55.750599 22876 fs_manager.cc:359] Metadata directory not provided
I20260812 05:22:55.750643 22876 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 05:22:55.750757 22876 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 05:22:55.751710 22876 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786512017613366-32144-0/minicluster-data/ts-0/data/instance:
uuid: "e183f30ebe8b419f8b2fdec556634e45"
format_stamp: "Formatted at 2026-08-12 05:22:55 on dist-test-slave-42z9"
server_key: "79fc4ee8ece2e5b458a42b62eebdf8e6"
server_key_iv: "ce475b6e38a19f92ff73166e94d397a9"
server_key_version: "encryptionkey@0"
I20260812 05:22:55.752012 22876 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786512017613366-32144-0/minicluster-data/ts-0/wal/instance:
uuid: "e183f30ebe8b419f8b2fdec556634e45"
format_stamp: "Formatted at 2026-08-12 05:22:55 on dist-test-slave-42z9"
server_key: "79fc4ee8ece2e5b458a42b62eebdf8e6"
server_key_iv: "ce475b6e38a19f92ff73166e94d397a9"
server_key_version: "encryptionkey@0"
I20260812 05:22:55.753235 22876 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260812 05:22:55.754071 22905 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 05:22:55.754248 22876 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260812 05:22:55.754343 22876 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786512017613366-32144-0/minicluster-data/ts-0/data,/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786512017613366-32144-0/minicluster-data/ts-0/wal
uuid: "e183f30ebe8b419f8b2fdec556634e45"
format_stamp: "Formatted at 2026-08-12 05:22:55 on dist-test-slave-42z9"
server_key: "79fc4ee8ece2e5b458a42b62eebdf8e6"
server_key_iv: "ce475b6e38a19f92ff73166e94d397a9"
server_key_version: "encryptionkey@0"
I20260812 05:22:55.754410 22876 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786512017613366-32144-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786512017613366-32144-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786512017613366-32144-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 05:22:55.763090 22876 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 05:22:55.764967 22876 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 05:22:55.765090 22876 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 05:22:55.765277 22876 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 05:22:55.765576 22876 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 05:22:55.765612 22876 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 05:22:55.765638 22876 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 05:22:55.765658 22876 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 05:22:55.770951 22876 rpc_server.cc:307] RPC server started. Bound to: 127.31.100.1:44501
I20260812 05:22:55.771013 23133 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.100.1:44501 every 8 connection(s)
I20260812 05:22:55.771273 22876 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786512017613366-32144-0/minicluster-data/ts-0/data/info.pb
I20260812 05:22:55.773011 32144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0waFvT/build/release/bin/kudu as pid 22876
I20260812 05:22:55.773120 32144 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786512017613366-32144-0/minicluster-data/ts-0/wal/instance
I20260812 05:22:55.773336 32144 external_mini_cluster.cc:1428] Setting key 53d664c2c6c8cf9e728e0148c497d2cc
Aug 12 05:22:55 dist-test-slave-42z9 krb5kdc[22682](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512175, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.100.1@KRBTEST.COM for kudu/127.31.100.62@KRBTEST.COM
I20260812 05:22:55.780432 22837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:22:55.771982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.100.1:52715 (local address 127.31.100.62:40577)
0812 05:22:55.772035 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0812 05:22:55.772037 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 05:22:55.772198 (+ 161us) server_negotiation.cc:407] Connection header received
0812 05:22:55.772701 (+ 503us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:22:55.772703 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:22:55.772715 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:22:55.772766 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:22:55.773917 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:22:55.774445 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:22:55.775721 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:22:55.776004 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:22:55.776062 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:22:55.778016 (+ 1954us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 05:22:55.778018 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 05:22:55.778018 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 05:22:55.778025 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 05:22:55.779230 (+ 1205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:22:55.779598 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:22:55.779599 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:22:55.779599 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:22:55.779649 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:22:55.779855 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:22:55.779855 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:22:55.779856 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:22:55.779957 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 05:22:55.780011 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0812 05:22:55.780257 (+ 246us) server_negotiation.cc:299] Negotiation successful
0812 05:22:55.780342 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260812 05:22:55.780570 23139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:22:55.771948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.62:40577 (local address 127.31.100.1:52715)
0812 05:22:55.772095 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0812 05:22:55.772129 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0812 05:22:55.772609 (+ 480us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:22:55.772829 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:22:55.772833 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:22:55.773592 (+ 759us) client_negotiation.cc:189] Negotiated authn=SASL
0812 05:22:55.773821 (+ 229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:22:55.773827 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:22:55.774528 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:22:55.774529 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:22:55.775635 (+ 1106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:22:55.775638 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:22:55.776079 (+ 441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:22:55.776080 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:22:55.776180 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:22:55.776580 (+ 400us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 05:22:55.776590 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0812 05:22:55.777934 (+ 1344us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 05:22:55.779300 (+ 1366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 05:22:55.779307 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 05:22:55.779309 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 05:22:55.779532 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 05:22:55.779703 (+ 171us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 05:22:55.779704 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 05:22:55.779704 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 05:22:55.779802 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 05:22:55.780010 (+ 208us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 05:22:55.780016 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 05:22:55.780145 (+ 129us) client_negotiation.cc:769] Sending connection context
0812 05:22:55.780229 (+ 84us) client_negotiation.cc:240] Negotiation successful
0812 05:22:55.780371 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":74,"threads_started":1}
I20260812 05:22:55.780962 23134 heartbeater.cc:347] Connected to a master server at 127.31.100.62:40577
I20260812 05:22:55.781059 23134 heartbeater.cc:464] Registering TS with master...
I20260812 05:22:55.781217 23134 heartbeater.cc:511] Master 127.31.100.62:40577 requested a full tablet report, sending...
I20260812 05:22:55.781710 22771 ts_manager.cc:194] Registered new tserver with Master: e183f30ebe8b419f8b2fdec556634e45 (127.31.100.1:44501)
WARNING: no policy specified for kudu/127.31.100.2@KRBTEST.COM; defaulting to no policy
I20260812 05:22:55.782310 22771 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.100.1@KRBTEST.COM'} at 127.31.100.1:52715
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.100.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.100.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.2.keytab.
Entry for principal kudu/127.31.100.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.2.keytab.
I20260812 05:22:55.798179 32144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0waFvT/build/release/bin/kudu
/tmp/dist-test-task0waFvT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786512017613366-32144-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786512017613366-32144-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786512017613366-32144-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786512017613366-32144-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.100.2:0
--local_ip_for_outbound_sockets=127.31.100.2
--webserver_interface=127.31.100.2
--webserver_port=0
--tserver_master_addrs=127.31.100.62:40577
--keytab_file=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.2.keytab
--principal=kudu/127.31.100.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.100.20:37413
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786512017613366-32144-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786512017613366-32144-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786512017613366-32144-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 05:22:55.877104 23142 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 05:22:55.877287 23142 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 05:22:55.877317 23142 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 05:22:55.878430 23142 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 05:22:55.878473 23142 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 05:22:55.878509 23142 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.100.2
I20260812 05:22:55.879938 23142 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.100.20:37413
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786512017613366-32144-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786512017613366-32144-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.2.keytab
--principal=kudu/127.31.100.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.100.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786512017613366-32144-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.100.2
--webserver_port=0
--tserver_master_addrs=127.31.100.62:40577
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23142
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.100.2
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786512017613366-32144-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 04:20:34 UTC on ccycloud.quasar-phqdqp.root.comops.site
build id 4581871
I20260812 05:22:55.880155 23142 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 05:22:55.880359 23142 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 05:22:55.885051 23150 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 05:22:55.885147 23155 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 05:22:55.885185 23142 server_base.cc:1034] running on GCE node
W20260812 05:22:55.885063 23149 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 05:22:55.885488 23142 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 05:22:55.885730 23142 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 05:22:55.886866 23142 hybrid_clock.cc:648] HybridClock initialized: now 1786512175886854 us; error 30 us; skew 500 ppm
Aug 12 05:22:55 dist-test-slave-42z9 krb5kdc[22682](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512175, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.100.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 05:22:55.889281 23142 init.cc:377] Logged in from keytab as kudu/127.31.100.2@KRBTEST.COM (short username kudu)
I20260812 05:22:55.890199 23142 webserver.cc:460] Webserver started at http://127.31.100.2:42543/ using document root <none> and password file <none>
I20260812 05:22:55.890409 23142 fs_manager.cc:359] Metadata directory not provided
I20260812 05:22:55.890451 23142 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 05:22:55.890566 23142 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 05:22:55.891520 23142 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786512017613366-32144-0/minicluster-data/ts-1/data/instance:
uuid: "323e33cb599a49dc8c4ad5ca105d6634"
format_stamp: "Formatted at 2026-08-12 05:22:55 on dist-test-slave-42z9"
server_key: "ec87d2ef04a1a1fe9b2bd7e62ab79afe"
server_key_iv: "673423269407aaced0bfb9334e1c7d54"
server_key_version: "encryptionkey@0"
I20260812 05:22:55.891870 23142 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786512017613366-32144-0/minicluster-data/ts-1/wal/instance:
uuid: "323e33cb599a49dc8c4ad5ca105d6634"
format_stamp: "Formatted at 2026-08-12 05:22:55 on dist-test-slave-42z9"
server_key: "ec87d2ef04a1a1fe9b2bd7e62ab79afe"
server_key_iv: "673423269407aaced0bfb9334e1c7d54"
server_key_version: "encryptionkey@0"
I20260812 05:22:55.893085 23142 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260812 05:22:55.893770 23170 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 05:22:55.893904 23142 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260812 05:22:55.893980 23142 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786512017613366-32144-0/minicluster-data/ts-1/data,/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786512017613366-32144-0/minicluster-data/ts-1/wal
uuid: "323e33cb599a49dc8c4ad5ca105d6634"
format_stamp: "Formatted at 2026-08-12 05:22:55 on dist-test-slave-42z9"
server_key: "ec87d2ef04a1a1fe9b2bd7e62ab79afe"
server_key_iv: "673423269407aaced0bfb9334e1c7d54"
server_key_version: "encryptionkey@0"
I20260812 05:22:55.894050 23142 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786512017613366-32144-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786512017613366-32144-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786512017613366-32144-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 05:22:55.900771 23142 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 05:22:55.902431 23142 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 05:22:55.902547 23142 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 05:22:55.902738 23142 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 05:22:55.903003 23142 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 05:22:55.903041 23142 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 05:22:55.903062 23142 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 05:22:55.903084 23142 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 05:22:55.908406 23142 rpc_server.cc:307] RPC server started. Bound to: 127.31.100.2:40255
I20260812 05:22:55.908442 23375 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.100.2:40255 every 8 connection(s)
I20260812 05:22:55.908705 23142 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786512017613366-32144-0/minicluster-data/ts-1/data/info.pb
I20260812 05:22:55.912508 32144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0waFvT/build/release/bin/kudu as pid 23142
I20260812 05:22:55.912616 32144 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786512017613366-32144-0/minicluster-data/ts-1/wal/instance
I20260812 05:22:55.912801 32144 external_mini_cluster.cc:1428] Setting key c6adf8c52e8b8bd4b101fdcc009db0d4
Aug 12 05:22:55 dist-test-slave-42z9 krb5kdc[22682](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512175, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.100.2@KRBTEST.COM for kudu/127.31.100.62@KRBTEST.COM
I20260812 05:22:55.917593 22837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:22:55.909317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.100.2:50957 (local address 127.31.100.62:40577)
0812 05:22:55.909370 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0812 05:22:55.909372 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 05:22:55.909580 (+ 208us) server_negotiation.cc:407] Connection header received
0812 05:22:55.910042 (+ 462us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:22:55.910043 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:22:55.910055 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:22:55.910096 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:22:55.910958 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:22:55.911452 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:22:55.912371 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:22:55.912580 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:22:55.912635 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:22:55.914998 (+ 2363us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 05:22:55.914999 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 05:22:55.914999 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 05:22:55.915006 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 05:22:55.916320 (+ 1314us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:22:55.916679 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:22:55.916681 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:22:55.916681 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:22:55.916742 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:22:55.916940 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:22:55.916941 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:22:55.916941 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:22:55.917067 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 05:22:55.917125 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0812 05:22:55.917395 (+ 270us) server_negotiation.cc:299] Negotiation successful
0812 05:22:55.917500 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260812 05:22:55.917696 23380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:22:55.909289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.62:40577 (local address 127.31.100.2:50957)
0812 05:22:55.909473 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0812 05:22:55.909507 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0812 05:22:55.909951 (+ 444us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:22:55.910169 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:22:55.910172 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:22:55.910696 (+ 524us) client_negotiation.cc:189] Negotiated authn=SASL
0812 05:22:55.910871 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:22:55.910875 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:22:55.911532 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:22:55.911533 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:22:55.912292 (+ 759us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:22:55.912294 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:22:55.912697 (+ 403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:22:55.912698 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:22:55.912804 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:22:55.913359 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 05:22:55.913368 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0812 05:22:55.914915 (+ 1547us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 05:22:55.916394 (+ 1479us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 05:22:55.916402 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 05:22:55.916404 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 05:22:55.916610 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 05:22:55.916800 (+ 190us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 05:22:55.916801 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 05:22:55.916801 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 05:22:55.916883 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 05:22:55.917126 (+ 243us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 05:22:55.917129 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 05:22:55.917274 (+ 145us) client_negotiation.cc:769] Sending connection context
0812 05:22:55.917362 (+ 88us) client_negotiation.cc:240] Negotiation successful
0812 05:22:55.917521 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":87,"threads_started":1}
I20260812 05:22:55.918139 23377 heartbeater.cc:347] Connected to a master server at 127.31.100.62:40577
I20260812 05:22:55.918217 23377 heartbeater.cc:464] Registering TS with master...
I20260812 05:22:55.918377 23377 heartbeater.cc:511] Master 127.31.100.62:40577 requested a full tablet report, sending...
I20260812 05:22:55.918748 22771 ts_manager.cc:194] Registered new tserver with Master: 323e33cb599a49dc8c4ad5ca105d6634 (127.31.100.2:40255)
I20260812 05:22:55.919220 22771 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.100.2@KRBTEST.COM'} at 127.31.100.2:50957
WARNING: no policy specified for kudu/127.31.100.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.100.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.100.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.3.keytab.
Entry for principal kudu/127.31.100.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.3.keytab.
I20260812 05:22:55.935606 32144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0waFvT/build/release/bin/kudu
/tmp/dist-test-task0waFvT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786512017613366-32144-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786512017613366-32144-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786512017613366-32144-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786512017613366-32144-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.100.3:0
--local_ip_for_outbound_sockets=127.31.100.3
--webserver_interface=127.31.100.3
--webserver_port=0
--tserver_master_addrs=127.31.100.62:40577
--keytab_file=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.3.keytab
--principal=kudu/127.31.100.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.100.20:37413
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786512017613366-32144-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786512017613366-32144-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786512017613366-32144-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 05:22:56.014232 23387 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 05:22:56.014417 23387 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 05:22:56.014438 23387 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 05:22:56.015504 23387 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 05:22:56.015539 23387 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 05:22:56.015564 23387 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.100.3
I20260812 05:22:56.016975 23387 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.100.20:37413
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786512017613366-32144-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786512017613366-32144-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.3.keytab
--principal=kudu/127.31.100.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.100.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786512017613366-32144-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.100.3
--webserver_port=0
--tserver_master_addrs=127.31.100.62:40577
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23387
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.100.3
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786512017613366-32144-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 04:20:34 UTC on ccycloud.quasar-phqdqp.root.comops.site
build id 4581871
I20260812 05:22:56.017176 23387 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 05:22:56.017395 23387 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 05:22:56.021999 23401 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 05:22:56.022006 23397 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 05:22:56.022025 23399 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 05:22:56.022238 23387 server_base.cc:1034] running on GCE node
I20260812 05:22:56.022428 23387 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 05:22:56.022655 23387 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 05:22:56.023790 23387 hybrid_clock.cc:648] HybridClock initialized: now 1786512176023775 us; error 26 us; skew 500 ppm
Aug 12 05:22:56 dist-test-slave-42z9 krb5kdc[22682](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512176, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.100.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 05:22:56.026144 23387 init.cc:377] Logged in from keytab as kudu/127.31.100.3@KRBTEST.COM (short username kudu)
I20260812 05:22:56.026544 23387 webserver.cc:460] Webserver started at http://127.31.100.3:38583/ using document root <none> and password file <none>
I20260812 05:22:56.026752 23387 fs_manager.cc:359] Metadata directory not provided
I20260812 05:22:56.026798 23387 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 05:22:56.026917 23387 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 05:22:56.027858 23387 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786512017613366-32144-0/minicluster-data/ts-2/data/instance:
uuid: "4d517dee1abe4f2e8070c046252ea7be"
format_stamp: "Formatted at 2026-08-12 05:22:56 on dist-test-slave-42z9"
server_key: "3b6ae4bd9db96d6907c558c9f73f404b"
server_key_iv: "44679731b0da20a67ac4afea59c62f29"
server_key_version: "encryptionkey@0"
I20260812 05:22:56.028173 23387 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786512017613366-32144-0/minicluster-data/ts-2/wal/instance:
uuid: "4d517dee1abe4f2e8070c046252ea7be"
format_stamp: "Formatted at 2026-08-12 05:22:56 on dist-test-slave-42z9"
server_key: "3b6ae4bd9db96d6907c558c9f73f404b"
server_key_iv: "44679731b0da20a67ac4afea59c62f29"
server_key_version: "encryptionkey@0"
I20260812 05:22:56.029425 23387 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260812 05:22:56.030144 23411 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 05:22:56.030287 23387 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260812 05:22:56.030369 23387 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786512017613366-32144-0/minicluster-data/ts-2/data,/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786512017613366-32144-0/minicluster-data/ts-2/wal
uuid: "4d517dee1abe4f2e8070c046252ea7be"
format_stamp: "Formatted at 2026-08-12 05:22:56 on dist-test-slave-42z9"
server_key: "3b6ae4bd9db96d6907c558c9f73f404b"
server_key_iv: "44679731b0da20a67ac4afea59c62f29"
server_key_version: "encryptionkey@0"
I20260812 05:22:56.030432 23387 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786512017613366-32144-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786512017613366-32144-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786512017613366-32144-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 05:22:56.039310 23387 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 05:22:56.040956 23387 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 05:22:56.041085 23387 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 05:22:56.041273 23387 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 05:22:56.041571 23387 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 05:22:56.041607 23387 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 05:22:56.041635 23387 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 05:22:56.041656 23387 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 05:22:56.047370 23387 rpc_server.cc:307] RPC server started. Bound to: 127.31.100.3:44797
I20260812 05:22:56.047415 23629 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.100.3:44797 every 8 connection(s)
I20260812 05:22:56.047694 23387 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786512017613366-32144-0/minicluster-data/ts-2/data/info.pb
I20260812 05:22:56.049747 32144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0waFvT/build/release/bin/kudu as pid 23387
I20260812 05:22:56.049867 32144 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786512017613366-32144-0/minicluster-data/ts-2/wal/instance
I20260812 05:22:56.050071 32144 external_mini_cluster.cc:1428] Setting key 1140ce97b79347432def72e3dd156a61
Aug 12 05:22:56 dist-test-slave-42z9 krb5kdc[22682](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512176, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.100.3@KRBTEST.COM for kudu/127.31.100.62@KRBTEST.COM
I20260812 05:22:56.056227 22837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:22:56.048353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.100.3:41641 (local address 127.31.100.62:40577)
0812 05:22:56.048401 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0812 05:22:56.048403 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 05:22:56.048602 (+ 199us) server_negotiation.cc:407] Connection header received
0812 05:22:56.049085 (+ 483us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:22:56.049086 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:22:56.049096 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:22:56.049146 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:22:56.050225 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:22:56.050972 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:22:56.051926 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:22:56.052114 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:22:56.052161 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:22:56.054173 (+ 2012us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 05:22:56.054175 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 05:22:56.054175 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 05:22:56.054181 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0812 05:22:56.055241 (+ 1060us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:22:56.055591 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:22:56.055591 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:22:56.055592 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:22:56.055636 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:22:56.055803 (+ 167us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:22:56.055803 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:22:56.055803 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:22:56.055888 (+ 85us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 05:22:56.055929 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0812 05:22:56.056115 (+ 186us) server_negotiation.cc:299] Negotiation successful
0812 05:22:56.056167 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260812 05:22:56.056399 23635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:22:56.048322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.62:40577 (local address 127.31.100.3:41641)
0812 05:22:56.048491 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0812 05:22:56.048528 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0812 05:22:56.048992 (+ 464us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:22:56.049196 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:22:56.049199 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:22:56.049874 (+ 675us) client_negotiation.cc:189] Negotiated authn=SASL
0812 05:22:56.050134 (+ 260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:22:56.050141 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:22:56.051064 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:22:56.051065 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:22:56.051860 (+ 795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:22:56.051863 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:22:56.052200 (+ 337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:22:56.052202 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:22:56.052309 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:22:56.052778 (+ 469us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 05:22:56.052785 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0812 05:22:56.054106 (+ 1321us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 05:22:56.055301 (+ 1195us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 05:22:56.055306 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 05:22:56.055308 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 05:22:56.055532 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 05:22:56.055678 (+ 146us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 05:22:56.055679 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 05:22:56.055679 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 05:22:56.055763 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 05:22:56.055926 (+ 163us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 05:22:56.055929 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 05:22:56.056028 (+ 99us) client_negotiation.cc:769] Sending connection context
0812 05:22:56.056100 (+ 72us) client_negotiation.cc:240] Negotiation successful
0812 05:22:56.056208 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":82,"threads_started":1}
I20260812 05:22:56.056818 23632 heartbeater.cc:347] Connected to a master server at 127.31.100.62:40577
I20260812 05:22:56.056900 23632 heartbeater.cc:464] Registering TS with master...
I20260812 05:22:56.057096 23632 heartbeater.cc:511] Master 127.31.100.62:40577 requested a full tablet report, sending...
I20260812 05:22:56.057451 22771 ts_manager.cc:194] Registered new tserver with Master: 4d517dee1abe4f2e8070c046252ea7be (127.31.100.3:44797)
I20260812 05:22:56.057828 22771 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.100.3@KRBTEST.COM'} at 127.31.100.3:41641
I20260812 05:22:56.061555 32144 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 05:22:56.067581 22837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:22:56.062767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59838 (local address 127.31.100.62:40577)
0812 05:22:56.062806 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0812 05:22:56.062808 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 05:22:56.062897 (+ 89us) server_negotiation.cc:407] Connection header received
0812 05:22:56.062936 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:22:56.062937 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:22:56.062944 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:22:56.062985 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:22:56.063463 (+ 478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:22:56.063908 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:22:56.064580 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:22:56.064741 (+ 161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:22:56.064782 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:22:56.065817 (+ 1035us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 05:22:56.065818 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 05:22:56.065818 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 05:22:56.065823 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0812 05:22:56.066786 (+ 963us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:22:56.067058 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:22:56.067059 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:22:56.067059 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:22:56.067097 (+ 38us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:22:56.067203 (+ 106us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:22:56.067204 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:22:56.067204 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:22:56.067317 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 05:22:56.067382 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0812 05:22:56.067459 (+ 77us) server_negotiation.cc:299] Negotiation successful
0812 05:22:56.067514 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260812 05:22:56.068953 22771 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59838:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 05:22:56.070464 22771 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 05:22:56.076056 23649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:22:56.072669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32888 (local address 127.31.100.2:40255)
0812 05:22:56.072849 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0812 05:22:56.072854 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 05:22:56.072882 (+ 28us) server_negotiation.cc:407] Connection header received
0812 05:22:56.072928 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:22:56.072930 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:22:56.072946 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:22:56.073036 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 05:22:56.073176 (+ 140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:22:56.073996 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:22:56.075032 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:22:56.075669 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:22:56.075730 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:22:56.075789 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0812 05:22:56.075939 (+ 150us) server_negotiation.cc:299] Negotiation successful
0812 05:22:56.075975 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":80,"threads_started":1}
I20260812 05:22:56.076066 23643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:22:56.072504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.2:40255 (local address 127.0.0.1:32888)
0812 05:22:56.072821 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0812 05:22:56.072829 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0812 05:22:56.072870 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:22:56.073045 (+ 175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:22:56.073049 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:22:56.073051 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 05:22:56.073115 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:22:56.073117 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:22:56.074093 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:22:56.074095 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:22:56.074943 (+ 848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:22:56.074946 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:22:56.075742 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:22:56.075743 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:22:56.075837 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:22:56.075849 (+ 12us) client_negotiation.cc:769] Sending connection context
0812 05:22:56.075915 (+ 66us) client_negotiation.cc:240] Negotiation successful
0812 05:22:56.075970 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":305,"mutex_wait_us":2,"thread_start_us":53,"threads_started":1}
I20260812 05:22:56.076587 23644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:22:56.072504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.3:44797 (local address 127.0.0.1:34032)
0812 05:22:56.072679 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0812 05:22:56.072718 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0812 05:22:56.072778 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:22:56.073330 (+ 552us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:22:56.073331 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:22:56.073333 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 05:22:56.073371 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:22:56.073372 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:22:56.074241 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:22:56.074242 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:22:56.075355 (+ 1113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:22:56.075358 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:22:56.076341 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:22:56.076342 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:22:56.076414 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:22:56.076417 (+ 3us) client_negotiation.cc:769] Sending connection context
0812 05:22:56.076476 (+ 59us) client_negotiation.cc:240] Negotiation successful
0812 05:22:56.076513 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"spinlock_wait_cycles":1408,"thread_start_us":113,"threads_started":1}
I20260812 05:22:56.076988 23266 tablet_service.cc:1467] Processing CreateTablet for tablet 1c4503a6bf6b45fa9f2e51edc3e83ce1 (DEFAULT_TABLE table=test-table [id=acedb8b609074c53b6f443b82e7a6a8d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 05:22:56.077385 23266 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c4503a6bf6b45fa9f2e51edc3e83ce1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 05:22:56.077642 23647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:22:56.072529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.1:44501 (local address 127.0.0.1:54840)
0812 05:22:56.072956 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0812 05:22:56.072966 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0812 05:22:56.073022 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:22:56.074104 (+ 1082us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:22:56.074104 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:22:56.074107 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 05:22:56.074143 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:22:56.074144 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:22:56.075497 (+ 1353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:22:56.075498 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:22:56.076526 (+ 1028us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:22:56.076529 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:22:56.077390 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:22:56.077391 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:22:56.077477 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:22:56.077482 (+ 5us) client_negotiation.cc:769] Sending connection context
0812 05:22:56.077537 (+ 55us) client_negotiation.cc:240] Negotiation successful
0812 05:22:56.077575 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":409,"thread_start_us":103,"threads_started":1}
I20260812 05:22:56.077795 23650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:22:56.072770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54840 (local address 127.31.100.1:44501)
0812 05:22:56.072949 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0812 05:22:56.072954 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 05:22:56.073012 (+ 58us) server_negotiation.cc:407] Connection header received
0812 05:22:56.073084 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:22:56.073086 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:22:56.073101 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:22:56.073183 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 05:22:56.074225 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:22:56.075403 (+ 1178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:22:56.076614 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:22:56.077306 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:22:56.077379 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:22:56.077433 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0812 05:22:56.077602 (+ 169us) server_negotiation.cc:299] Negotiation successful
0812 05:22:56.077646 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":90,"threads_started":1}
I20260812 05:22:56.078388 23003 tablet_service.cc:1467] Processing CreateTablet for tablet 1c4503a6bf6b45fa9f2e51edc3e83ce1 (DEFAULT_TABLE table=test-table [id=acedb8b609074c53b6f443b82e7a6a8d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 05:22:56.078685 23003 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c4503a6bf6b45fa9f2e51edc3e83ce1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 05:22:56.080060 23651 tablet_bootstrap.cc:492] T 1c4503a6bf6b45fa9f2e51edc3e83ce1 P 323e33cb599a49dc8c4ad5ca105d6634: Bootstrap starting.
I20260812 05:22:56.080108 23648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:22:56.072667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34032 (local address 127.31.100.3:44797)
0812 05:22:56.073142 (+ 475us) server_negotiation.cc:206] Beginning negotiation
0812 05:22:56.073146 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 05:22:56.073154 (+ 8us) server_negotiation.cc:407] Connection header received
0812 05:22:56.073187 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:22:56.073189 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:22:56.073202 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:22:56.073281 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 05:22:56.073427 (+ 146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:22:56.074133 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:22:56.075463 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:22:56.076269 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:22:56.076331 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:22:56.076380 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0812 05:22:56.076494 (+ 114us) server_negotiation.cc:299] Negotiation successful
0812 05:22:56.076528 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":436,"thread_start_us":78,"threads_started":1}
I20260812 05:22:56.080686 23494 tablet_service.cc:1467] Processing CreateTablet for tablet 1c4503a6bf6b45fa9f2e51edc3e83ce1 (DEFAULT_TABLE table=test-table [id=acedb8b609074c53b6f443b82e7a6a8d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 05:22:56.080807 23651 tablet_bootstrap.cc:654] T 1c4503a6bf6b45fa9f2e51edc3e83ce1 P 323e33cb599a49dc8c4ad5ca105d6634: Neither blocks nor log segments found. Creating new log.
I20260812 05:22:56.081019 23494 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c4503a6bf6b45fa9f2e51edc3e83ce1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 05:22:56.081343 23651 log.cc:824] T 1c4503a6bf6b45fa9f2e51edc3e83ce1 P 323e33cb599a49dc8c4ad5ca105d6634: Log is configured to *not* fsync() on all Append() calls
I20260812 05:22:56.081359 23653 tablet_bootstrap.cc:492] T 1c4503a6bf6b45fa9f2e51edc3e83ce1 P e183f30ebe8b419f8b2fdec556634e45: Bootstrap starting.
I20260812 05:22:56.082093 23653 tablet_bootstrap.cc:654] T 1c4503a6bf6b45fa9f2e51edc3e83ce1 P e183f30ebe8b419f8b2fdec556634e45: Neither blocks nor log segments found. Creating new log.
I20260812 05:22:56.082587 23653 log.cc:824] T 1c4503a6bf6b45fa9f2e51edc3e83ce1 P e183f30ebe8b419f8b2fdec556634e45: Log is configured to *not* fsync() on all Append() calls
I20260812 05:22:56.083256 23653 tablet_bootstrap.cc:492] T 1c4503a6bf6b45fa9f2e51edc3e83ce1 P e183f30ebe8b419f8b2fdec556634e45: No bootstrap required, opened a new log
I20260812 05:22:56.083317 23653 ts_tablet_manager.cc:1397] T 1c4503a6bf6b45fa9f2e51edc3e83ce1 P e183f30ebe8b419f8b2fdec556634e45: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260812 05:22:56.083604 23657 tablet_bootstrap.cc:492] T 1c4503a6bf6b45fa9f2e51edc3e83ce1 P 4d517dee1abe4f2e8070c046252ea7be: Bootstrap starting.
I20260812 05:22:56.084235 23657 tablet_bootstrap.cc:654] T 1c4503a6bf6b45fa9f2e51edc3e83ce1 P 4d517dee1abe4f2e8070c046252ea7be: Neither blocks nor log segments found. Creating new log.
I20260812 05:22:56.084240 23651 tablet_bootstrap.cc:492] T 1c4503a6bf6b45fa9f2e51edc3e83ce1 P 323e33cb599a49dc8c4ad5ca105d6634: No bootstrap required, opened a new log
I20260812 05:22:56.084307 23651 ts_tablet_manager.cc:1397] T 1c4503a6bf6b45fa9f2e51edc3e83ce1 P 323e33cb599a49dc8c4ad5ca105d6634: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260812 05:22:56.084762 23657 log.cc:824] T 1c4503a6bf6b45fa9f2e51edc3e83ce1 P 4d517dee1abe4f2e8070c046252ea7be: Log is configured to *not* fsync() on all Append() calls
I20260812 05:22:56.085064 23653 raft_consensus.cc:348] T 1c4503a6bf6b45fa9f2e51edc3e83ce1 P e183f30ebe8b419f8b2fdec556634e45 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "323e33cb599a49dc8c4ad5ca105d6634" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 40255 } } peers { permanent_uuid: "e183f30ebe8b419f8b2fdec556634e45" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 44501 } } peers { permanent_uuid: "4d517dee1abe4f2e8070c046252ea7be" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 44797 } }
I20260812 05:22:56.085238 23653 raft_consensus.cc:374] T 1c4503a6bf6b45fa9f2e51edc3e83ce1 P e183f30ebe8b419f8b2fdec556634e45 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 05:22:56.085282 23653 raft_consensus.cc:729] T 1c4503a6bf6b45fa9f2e51edc3e83ce1 P e183f30ebe8b419f8b2fdec556634e45 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e183f30ebe8b419f8b2fdec556634e45, State: Initialized, Role: FOLLOWER
I20260812 05:22:56.085392 23653 consensus_queue.cc:260] T 1c4503a6bf6b45fa9f2e51edc3e83ce1 P e183f30ebe8b419f8b2fdec556634e45 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "323e33cb599a49dc8c4ad5ca105d6634" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 40255 } } peers { permanent_uuid: "e183f30ebe8b419f8b2fdec556634e45" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 44501 } } peers { permanent_uuid: "4d517dee1abe4f2e8070c046252ea7be" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 44797 } }
I20260812 05:22:56.085585 23657 tablet_bootstrap.cc:492] T 1c4503a6bf6b45fa9f2e51edc3e83ce1 P 4d517dee1abe4f2e8070c046252ea7be: No bootstrap required, opened a new log
I20260812 05:22:56.085654 23657 ts_tablet_manager.cc:1397] T 1c4503a6bf6b45fa9f2e51edc3e83ce1 P 4d517dee1abe4f2e8070c046252ea7be: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260812 05:22:56.085693 23653 ts_tablet_manager.cc:1428] T 1c4503a6bf6b45fa9f2e51edc3e83ce1 P e183f30ebe8b419f8b2fdec556634e45: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260812 05:22:56.085835 23134 heartbeater.cc:503] Master 127.31.100.62:40577 was elected leader, sending a full tablet report...
I20260812 05:22:56.086190 23651 raft_consensus.cc:348] T 1c4503a6bf6b45fa9f2e51edc3e83ce1 P 323e33cb599a49dc8c4ad5ca105d6634 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "323e33cb599a49dc8c4ad5ca105d6634" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 40255 } } peers { permanent_uuid: "e183f30ebe8b419f8b2fdec556634e45" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 44501 } } peers { permanent_uuid: "4d517dee1abe4f2e8070c046252ea7be" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 44797 } }
I20260812 05:22:56.086344 23651 raft_consensus.cc:374] T 1c4503a6bf6b45fa9f2e51edc3e83ce1 P 323e33cb599a49dc8c4ad5ca105d6634 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 05:22:56.086380 23651 raft_consensus.cc:729] T 1c4503a6bf6b45fa9f2e51edc3e83ce1 P 323e33cb599a49dc8c4ad5ca105d6634 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 323e33cb599a49dc8c4ad5ca105d6634, State: Initialized, Role: FOLLOWER
I20260812 05:22:56.086493 23651 consensus_queue.cc:260] T 1c4503a6bf6b45fa9f2e51edc3e83ce1 P 323e33cb599a49dc8c4ad5ca105d6634 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "323e33cb599a49dc8c4ad5ca105d6634" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 40255 } } peers { permanent_uuid: "e183f30ebe8b419f8b2fdec556634e45" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 44501 } } peers { permanent_uuid: "4d517dee1abe4f2e8070c046252ea7be" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 44797 } }
I20260812 05:22:56.086771 23651 ts_tablet_manager.cc:1428] T 1c4503a6bf6b45fa9f2e51edc3e83ce1 P 323e33cb599a49dc8c4ad5ca105d6634: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260812 05:22:56.086792 23377 heartbeater.cc:503] Master 127.31.100.62:40577 was elected leader, sending a full tablet report...
I20260812 05:22:56.087355 23657 raft_consensus.cc:348] T 1c4503a6bf6b45fa9f2e51edc3e83ce1 P 4d517dee1abe4f2e8070c046252ea7be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "323e33cb599a49dc8c4ad5ca105d6634" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 40255 } } peers { permanent_uuid: "e183f30ebe8b419f8b2fdec556634e45" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 44501 } } peers { permanent_uuid: "4d517dee1abe4f2e8070c046252ea7be" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 44797 } }
I20260812 05:22:56.087471 23657 raft_consensus.cc:374] T 1c4503a6bf6b45fa9f2e51edc3e83ce1 P 4d517dee1abe4f2e8070c046252ea7be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 05:22:56.087491 23657 raft_consensus.cc:729] T 1c4503a6bf6b45fa9f2e51edc3e83ce1 P 4d517dee1abe4f2e8070c046252ea7be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d517dee1abe4f2e8070c046252ea7be, State: Initialized, Role: FOLLOWER
I20260812 05:22:56.087592 23657 consensus_queue.cc:260] T 1c4503a6bf6b45fa9f2e51edc3e83ce1 P 4d517dee1abe4f2e8070c046252ea7be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "323e33cb599a49dc8c4ad5ca105d6634" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 40255 } } peers { permanent_uuid: "e183f30ebe8b419f8b2fdec556634e45" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 44501 } } peers { permanent_uuid: "4d517dee1abe4f2e8070c046252ea7be" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 44797 } }
I20260812 05:22:56.087827 23657 ts_tablet_manager.cc:1428] T 1c4503a6bf6b45fa9f2e51edc3e83ce1 P 4d517dee1abe4f2e8070c046252ea7be: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260812 05:22:56.087867 23632 heartbeater.cc:503] Master 127.31.100.62:40577 was elected leader, sending a full tablet report...
W20260812 05:22:56.159209 23379 tablet.cc:2369] T 1c4503a6bf6b45fa9f2e51edc3e83ce1 P 323e33cb599a49dc8c4ad5ca105d6634: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 05:22:56.271953 23138 tablet.cc:2369] T 1c4503a6bf6b45fa9f2e51edc3e83ce1 P e183f30ebe8b419f8b2fdec556634e45: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 05:22:56.283639 23660 raft_consensus.cc:482] T 1c4503a6bf6b45fa9f2e51edc3e83ce1 P 323e33cb599a49dc8c4ad5ca105d6634 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 05:22:56.283768 23660 raft_consensus.cc:504] T 1c4503a6bf6b45fa9f2e51edc3e83ce1 P 323e33cb599a49dc8c4ad5ca105d6634 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "323e33cb599a49dc8c4ad5ca105d6634" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 40255 } } peers { permanent_uuid: "e183f30ebe8b419f8b2fdec556634e45" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 44501 } } peers { permanent_uuid: "4d517dee1abe4f2e8070c046252ea7be" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 44797 } }
I20260812 05:22:56.284124 23660 leader_election.cc:288] T 1c4503a6bf6b45fa9f2e51edc3e83ce1 P 323e33cb599a49dc8c4ad5ca105d6634 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e183f30ebe8b419f8b2fdec556634e45 (127.31.100.1:44501), 4d517dee1abe4f2e8070c046252ea7be (127.31.100.3:44797)
I20260812 05:22:56.287917 23650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:22:56.284381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.100.2:45571 (local address 127.31.100.1:44501)
0812 05:22:56.284421 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0812 05:22:56.284423 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 05:22:56.284568 (+ 145us) server_negotiation.cc:407] Connection header received
0812 05:22:56.284644 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:22:56.284646 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:22:56.284653 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:22:56.284694 (+ 41us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 05:22:56.284818 (+ 124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:22:56.285538 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:22:56.287010 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:22:56.287578 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:22:56.287651 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:22:56.287660 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0812 05:22:56.287809 (+ 149us) server_negotiation.cc:299] Negotiation successful
0812 05:22:56.287846 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260812 05:22:56.287917 23648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:22:56.284399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.100.2:50295 (local address 127.31.100.3:44797)
0812 05:22:56.284458 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0812 05:22:56.284460 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 05:22:56.284469 (+ 9us) server_negotiation.cc:407] Connection header received
0812 05:22:56.284499 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:22:56.284500 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:22:56.284510 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:22:56.284563 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 05:22:56.284739 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:22:56.285528 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:22:56.287007 (+ 1479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:22:56.287586 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:22:56.287651 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:22:56.287660 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0812 05:22:56.287809 (+ 149us) server_negotiation.cc:299] Negotiation successful
0812 05:22:56.287846 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260812 05:22:56.287938 23380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:22:56.284303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.3:44797 (local address 127.31.100.2:50295)
0812 05:22:56.284360 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0812 05:22:56.284371 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0812 05:22:56.284413 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:22:56.284601 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:22:56.284602 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:22:56.284604 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 05:22:56.284672 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:22:56.284673 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:22:56.285597 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:22:56.285598 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:22:56.286899 (+ 1301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:22:56.286903 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:22:56.287649 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:22:56.287650 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:22:56.287745 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:22:56.287747 (+ 2us) client_negotiation.cc:769] Sending connection context
0812 05:22:56.287799 (+ 52us) client_negotiation.cc:240] Negotiation successful
0812 05:22:56.287850 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38,"mutex_wait_us":70}
I20260812 05:22:56.287945 23663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:22:56.284310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.1:44501 (local address 127.31.100.2:45571)
0812 05:22:56.284493 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0812 05:22:56.284501 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0812 05:22:56.284567 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:22:56.284708 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:22:56.284708 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:22:56.284710 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 05:22:56.284763 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:22:56.284764 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:22:56.285597 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:22:56.285598 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:22:56.286899 (+ 1301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:22:56.286903 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:22:56.287649 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:22:56.287650 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:22:56.287740 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:22:56.287744 (+ 4us) client_negotiation.cc:769] Sending connection context
0812 05:22:56.287799 (+ 55us) client_negotiation.cc:240] Negotiation successful
0812 05:22:56.287850 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":170,"mutex_wait_us":64,"thread_start_us":99,"threads_started":1}
I20260812 05:22:56.288331 23537 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c4503a6bf6b45fa9f2e51edc3e83ce1" candidate_uuid: "323e33cb599a49dc8c4ad5ca105d6634" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d517dee1abe4f2e8070c046252ea7be" is_pre_election: true
I20260812 05:22:56.288352 23046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c4503a6bf6b45fa9f2e51edc3e83ce1" candidate_uuid: "323e33cb599a49dc8c4ad5ca105d6634" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e183f30ebe8b419f8b2fdec556634e45" is_pre_election: true
I20260812 05:22:56.288488 23537 raft_consensus.cc:2454] T 1c4503a6bf6b45fa9f2e51edc3e83ce1 P 4d517dee1abe4f2e8070c046252ea7be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 323e33cb599a49dc8c4ad5ca105d6634 in term 0.
I20260812 05:22:56.288507 23046 raft_consensus.cc:2454] T 1c4503a6bf6b45fa9f2e51edc3e83ce1 P e183f30ebe8b419f8b2fdec556634e45 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 323e33cb599a49dc8c4ad5ca105d6634 in term 0.
I20260812 05:22:56.288714 23179 leader_election.cc:302] T 1c4503a6bf6b45fa9f2e51edc3e83ce1 P 323e33cb599a49dc8c4ad5ca105d6634 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 323e33cb599a49dc8c4ad5ca105d6634, e183f30ebe8b419f8b2fdec556634e45; no voters:
I20260812 05:22:56.288829 23660 raft_consensus.cc:2781] T 1c4503a6bf6b45fa9f2e51edc3e83ce1 P 323e33cb599a49dc8c4ad5ca105d6634 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 05:22:56.288887 23660 raft_consensus.cc:482] T 1c4503a6bf6b45fa9f2e51edc3e83ce1 P 323e33cb599a49dc8c4ad5ca105d6634 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 05:22:56.288903 23660 raft_consensus.cc:3037] T 1c4503a6bf6b45fa9f2e51edc3e83ce1 P 323e33cb599a49dc8c4ad5ca105d6634 [term 0 FOLLOWER]: Advancing to term 1
I20260812 05:22:56.290268 23660 raft_consensus.cc:504] T 1c4503a6bf6b45fa9f2e51edc3e83ce1 P 323e33cb599a49dc8c4ad5ca105d6634 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "323e33cb599a49dc8c4ad5ca105d6634" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 40255 } } peers { permanent_uuid: "e183f30ebe8b419f8b2fdec556634e45" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 44501 } } peers { permanent_uuid: "4d517dee1abe4f2e8070c046252ea7be" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 44797 } }
I20260812 05:22:56.290441 23660 leader_election.cc:288] T 1c4503a6bf6b45fa9f2e51edc3e83ce1 P 323e33cb599a49dc8c4ad5ca105d6634 [CANDIDATE]: Term 1 election: Requested vote from peers e183f30ebe8b419f8b2fdec556634e45 (127.31.100.1:44501), 4d517dee1abe4f2e8070c046252ea7be (127.31.100.3:44797)
I20260812 05:22:56.290628 23046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c4503a6bf6b45fa9f2e51edc3e83ce1" candidate_uuid: "323e33cb599a49dc8c4ad5ca105d6634" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e183f30ebe8b419f8b2fdec556634e45"
I20260812 05:22:56.290671 23537 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c4503a6bf6b45fa9f2e51edc3e83ce1" candidate_uuid: "323e33cb599a49dc8c4ad5ca105d6634" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d517dee1abe4f2e8070c046252ea7be"
I20260812 05:22:56.290702 23046 raft_consensus.cc:3037] T 1c4503a6bf6b45fa9f2e51edc3e83ce1 P e183f30ebe8b419f8b2fdec556634e45 [term 0 FOLLOWER]: Advancing to term 1
I20260812 05:22:56.290756 23537 raft_consensus.cc:3037] T 1c4503a6bf6b45fa9f2e51edc3e83ce1 P 4d517dee1abe4f2e8070c046252ea7be [term 0 FOLLOWER]: Advancing to term 1
I20260812 05:22:56.291424 23046 raft_consensus.cc:2454] T 1c4503a6bf6b45fa9f2e51edc3e83ce1 P e183f30ebe8b419f8b2fdec556634e45 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 323e33cb599a49dc8c4ad5ca105d6634 in term 1.
I20260812 05:22:56.291683 23179 leader_election.cc:302] T 1c4503a6bf6b45fa9f2e51edc3e83ce1 P 323e33cb599a49dc8c4ad5ca105d6634 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 323e33cb599a49dc8c4ad5ca105d6634, e183f30ebe8b419f8b2fdec556634e45; no voters:
I20260812 05:22:56.291795 23660 raft_consensus.cc:2781] T 1c4503a6bf6b45fa9f2e51edc3e83ce1 P 323e33cb599a49dc8c4ad5ca105d6634 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 05:22:56.291941 23660 raft_consensus.cc:686] T 1c4503a6bf6b45fa9f2e51edc3e83ce1 P 323e33cb599a49dc8c4ad5ca105d6634 [term 1 LEADER]: Becoming Leader. State: Replica: 323e33cb599a49dc8c4ad5ca105d6634, State: Running, Role: LEADER
I20260812 05:22:56.292038 23660 consensus_queue.cc:237] T 1c4503a6bf6b45fa9f2e51edc3e83ce1 P 323e33cb599a49dc8c4ad5ca105d6634 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "323e33cb599a49dc8c4ad5ca105d6634" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 40255 } } peers { permanent_uuid: "e183f30ebe8b419f8b2fdec556634e45" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 44501 } } peers { permanent_uuid: "4d517dee1abe4f2e8070c046252ea7be" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 44797 } }
I20260812 05:22:56.292316 23537 raft_consensus.cc:2454] T 1c4503a6bf6b45fa9f2e51edc3e83ce1 P 4d517dee1abe4f2e8070c046252ea7be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 323e33cb599a49dc8c4ad5ca105d6634 in term 1.
I20260812 05:22:56.292835 22768 catalog_manager.cc:5495] T 1c4503a6bf6b45fa9f2e51edc3e83ce1 P 323e33cb599a49dc8c4ad5ca105d6634 reported cstate change: term changed from 0 to 1, leader changed from <none> to 323e33cb599a49dc8c4ad5ca105d6634 (127.31.100.2). New cstate: current_term: 1 leader_uuid: "323e33cb599a49dc8c4ad5ca105d6634" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "323e33cb599a49dc8c4ad5ca105d6634" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 40255 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e183f30ebe8b419f8b2fdec556634e45" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 44501 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4d517dee1abe4f2e8070c046252ea7be" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 44797 } health_report { overall_health: UNKNOWN } } }
W20260812 05:22:56.298204 23634 tablet.cc:2369] T 1c4503a6bf6b45fa9f2e51edc3e83ce1 P 4d517dee1abe4f2e8070c046252ea7be: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 05:22:56.299805 23649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:22:56.297615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32890 (local address 127.31.100.2:40255)
0812 05:22:56.297651 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0812 05:22:56.297653 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 05:22:56.297678 (+ 25us) server_negotiation.cc:407] Connection header received
0812 05:22:56.297734 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:22:56.297735 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:22:56.297749 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:22:56.297806 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 05:22:56.297884 (+ 78us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:22:56.298460 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:22:56.299176 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:22:56.299356 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:22:56.299427 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:22:56.299525 (+ 98us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 05:22:56.299609 (+ 84us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 05:22:56.299660 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0812 05:22:56.299729 (+ 69us) server_negotiation.cc:299] Negotiation successful
0812 05:22:56.299745 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260812 05:22:56.301340 23537 raft_consensus.cc:1261] T 1c4503a6bf6b45fa9f2e51edc3e83ce1 P 4d517dee1abe4f2e8070c046252ea7be [term 1 FOLLOWER]: Refusing update from remote peer 323e33cb599a49dc8c4ad5ca105d6634: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 05:22:56.301352 23046 raft_consensus.cc:1261] T 1c4503a6bf6b45fa9f2e51edc3e83ce1 P e183f30ebe8b419f8b2fdec556634e45 [term 1 FOLLOWER]: Refusing update from remote peer 323e33cb599a49dc8c4ad5ca105d6634: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 05:22:56.301657 23665 consensus_queue.cc:1035] T 1c4503a6bf6b45fa9f2e51edc3e83ce1 P 323e33cb599a49dc8c4ad5ca105d6634 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d517dee1abe4f2e8070c046252ea7be" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 44797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 05:22:56.301769 23660 consensus_queue.cc:1035] T 1c4503a6bf6b45fa9f2e51edc3e83ce1 P 323e33cb599a49dc8c4ad5ca105d6634 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e183f30ebe8b419f8b2fdec556634e45" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 44501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 05:22:56.304291 23675 mvcc.cc:205] Tried to move back new op lower bound from 7317553874128048128 to 7317553874092658688. Current Snapshot: MvccSnapshot[applied={T|T < 7317553874128048128}]
I20260812 05:22:56.304648 23674 mvcc.cc:205] Tried to move back new op lower bound from 7317553874128048128 to 7317553874092658688. Current Snapshot: MvccSnapshot[applied={T|T < 7317553874128048128}]
I20260812 05:22:56.306058 22768 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59838:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 05:22:56.306133 22768 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59838:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 05:22:56.306790 22768 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c866d37f66fb4b64bae37566f2baad8e: Sending DeleteTablet for 3 replicas of tablet 1c4503a6bf6b45fa9f2e51edc3e83ce1
I20260812 05:22:56.307111 23003 tablet_service.cc:1514] Processing DeleteTablet for tablet 1c4503a6bf6b45fa9f2e51edc3e83ce1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 05:22:56 UTC) from {username='kudu', principal='kudu/127.31.100.62@KRBTEST.COM'} at 127.0.0.1:54840
I20260812 05:22:56.307130 23494 tablet_service.cc:1514] Processing DeleteTablet for tablet 1c4503a6bf6b45fa9f2e51edc3e83ce1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 05:22:56 UTC) from {username='kudu', principal='kudu/127.31.100.62@KRBTEST.COM'} at 127.0.0.1:34032
I20260812 05:22:56.307183 32144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0waFvT/build/release/bin/kudu with pid 22876
I20260812 05:22:56.311676 23686 tablet_replica.cc:330] T 1c4503a6bf6b45fa9f2e51edc3e83ce1 P 4d517dee1abe4f2e8070c046252ea7be: stopping tablet replica
I20260812 05:22:56.311805 23686 raft_consensus.cc:2229] T 1c4503a6bf6b45fa9f2e51edc3e83ce1 P 4d517dee1abe4f2e8070c046252ea7be [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 05:22:56.311931 23686 raft_consensus.cc:2258] T 1c4503a6bf6b45fa9f2e51edc3e83ce1 P 4d517dee1abe4f2e8070c046252ea7be [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 05:22:56.314106 23686 ts_tablet_manager.cc:1905] T 1c4503a6bf6b45fa9f2e51edc3e83ce1 P 4d517dee1abe4f2e8070c046252ea7be: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 05:22:56.315505 23686 ts_tablet_manager.cc:1918] T 1c4503a6bf6b45fa9f2e51edc3e83ce1 P 4d517dee1abe4f2e8070c046252ea7be: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 05:22:56.315567 23686 log.cc:1196] T 1c4503a6bf6b45fa9f2e51edc3e83ce1 P 4d517dee1abe4f2e8070c046252ea7be: Deleting WAL directory at /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786512017613366-32144-0/minicluster-data/ts-2/wal/wals/1c4503a6bf6b45fa9f2e51edc3e83ce1
I20260812 05:22:56.315645 23266 tablet_service.cc:1514] Processing DeleteTablet for tablet 1c4503a6bf6b45fa9f2e51edc3e83ce1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 05:22:56 UTC) from {username='kudu', principal='kudu/127.31.100.62@KRBTEST.COM'} at 127.0.0.1:32888
I20260812 05:22:56.315819 23686 ts_tablet_manager.cc:1939] T 1c4503a6bf6b45fa9f2e51edc3e83ce1 P 4d517dee1abe4f2e8070c046252ea7be: Deleting consensus metadata
I20260812 05:22:56.316876 23690 tablet_replica.cc:330] T 1c4503a6bf6b45fa9f2e51edc3e83ce1 P 323e33cb599a49dc8c4ad5ca105d6634: stopping tablet replica
I20260812 05:22:56.316987 23690 raft_consensus.cc:2229] T 1c4503a6bf6b45fa9f2e51edc3e83ce1 P 323e33cb599a49dc8c4ad5ca105d6634 [term 1 LEADER]: Raft consensus shutting down.
I20260812 05:22:56.317116 23690 raft_consensus.cc:2258] T 1c4503a6bf6b45fa9f2e51edc3e83ce1 P 323e33cb599a49dc8c4ad5ca105d6634 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 05:22:56.317576 23690 ts_tablet_manager.cc:1905] T 1c4503a6bf6b45fa9f2e51edc3e83ce1 P 323e33cb599a49dc8c4ad5ca105d6634: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 05:22:56.318655 22735 catalog_manager.cc:4841] TS 4d517dee1abe4f2e8070c046252ea7be (127.31.100.3:44797): tablet 1c4503a6bf6b45fa9f2e51edc3e83ce1 (table test-table [id=acedb8b609074c53b6f443b82e7a6a8d]) successfully deleted
I20260812 05:22:56.319098 23690 ts_tablet_manager.cc:1918] T 1c4503a6bf6b45fa9f2e51edc3e83ce1 P 323e33cb599a49dc8c4ad5ca105d6634: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 05:22:56.319172 23690 log.cc:1196] T 1c4503a6bf6b45fa9f2e51edc3e83ce1 P 323e33cb599a49dc8c4ad5ca105d6634: Deleting WAL directory at /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786512017613366-32144-0/minicluster-data/ts-1/wal/wals/1c4503a6bf6b45fa9f2e51edc3e83ce1
I20260812 05:22:56.319448 23690 ts_tablet_manager.cc:1939] T 1c4503a6bf6b45fa9f2e51edc3e83ce1 P 323e33cb599a49dc8c4ad5ca105d6634: Deleting consensus metadata
I20260812 05:22:56.319820 22736 catalog_manager.cc:4841] TS 323e33cb599a49dc8c4ad5ca105d6634 (127.31.100.2:40255): tablet 1c4503a6bf6b45fa9f2e51edc3e83ce1 (table test-table [id=acedb8b609074c53b6f443b82e7a6a8d]) successfully deleted
W20260812 05:22:56.323657 22737 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.31.100.1:44501 (error 108)
I20260812 05:22:56.323906 32144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0waFvT/build/release/bin/kudu with pid 23142
I20260812 05:22:56.325387 23647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:22:56.324012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.1:44501 (local address 127.0.0.1:54852)
0812 05:22:56.325312 (+ 1300us) negotiation.cc:107] Waiting for socket to connect
0812 05:22:56.325356 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.100.1:44501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1276}
W20260812 05:22:56.328732 22737 catalog_manager.cc:4584] TS e183f30ebe8b419f8b2fdec556634e45 (127.31.100.1:44501): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1c4503a6bf6b45fa9f2e51edc3e83ce1: Network error: Client connection negotiation failed: client connection to 127.31.100.1:44501: connect: Connection refused (error 111)
I20260812 05:22:56.338778 32144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0waFvT/build/release/bin/kudu with pid 23387
I20260812 05:22:56.349514 32144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0waFvT/build/release/bin/kudu with pid 22705
I20260812 05:22:56.360311 32144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0waFvT/build/release/bin/kudu
/tmp/dist-test-task0waFvT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786512017613366-32144-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786512017613366-32144-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786512017613366-32144-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786512017613366-32144-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.100.62:40577
--webserver_interface=127.31.100.62
--webserver_port=43853
--keytab_file=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.62.keytab
--principal=kudu/127.31.100.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.100.20:37413
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.100.62:40577
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786512017613366-32144-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786512017613366-32144-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786512017613366-32144-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 05:22:56.442273 23692 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 05:22:56.442478 23692 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 05:22:56.442499 23692 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 05:22:56.443624 23692 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 05:22:56.443663 23692 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 05:22:56.443676 23692 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 05:22:56.443687 23692 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 05:22:56.443698 23692 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 05:22:56.445166 23692 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.100.20:37413
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786512017613366-32144-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786512017613366-32144-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.31.100.62:40577
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.62.keytab
--principal=kudu/127.31.100.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.100.62:40577
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786512017613366-32144-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.100.62
--webserver_port=43853
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23692
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786512017613366-32144-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 04:20:34 UTC on ccycloud.quasar-phqdqp.root.comops.site
build id 4581871
I20260812 05:22:56.445400 23692 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 05:22:56.445639 23692 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 05:22:56.454818 23708 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 05:22:56.454830 23703 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 05:22:56.454804 23705 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 05:22:56.455061 23692 server_base.cc:1034] running on GCE node
I20260812 05:22:56.455322 23692 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 05:22:56.455629 23692 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 05:22:56.456770 23692 hybrid_clock.cc:648] HybridClock initialized: now 1786512176456753 us; error 32 us; skew 500 ppm
Aug 12 05:22:56 dist-test-slave-42z9 krb5kdc[22682](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512176, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.100.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 05:22:56.461982 23692 init.cc:377] Logged in from keytab as kudu/127.31.100.62@KRBTEST.COM (short username kudu)
I20260812 05:22:56.462457 23692 webserver.cc:460] Webserver started at http://127.31.100.62:43853/ using document root <none> and password file <none>
I20260812 05:22:56.462684 23692 fs_manager.cc:359] Metadata directory not provided
I20260812 05:22:56.462750 23692 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 05:22:56.464102 23692 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260812 05:22:56.464700 23720 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 05:22:56.464864 23692 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260812 05:22:56.464951 23692 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786512017613366-32144-0/minicluster-data/master-0/data,/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786512017613366-32144-0/minicluster-data/master-0/wal
uuid: "c866d37f66fb4b64bae37566f2baad8e"
format_stamp: "Formatted at 2026-08-12 05:22:55 on dist-test-slave-42z9"
server_key: "9d146bbc124be7926356b761a88170c3"
server_key_iv: "59d13cc9c7748afda4d9392b9aad4348"
server_key_version: "encryptionkey@0"
I20260812 05:22:56.465343 23692 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786512017613366-32144-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786512017613366-32144-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786512017613366-32144-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 05:22:56.487542 23692 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 05:22:56.489288 23692 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 05:22:56.489426 23692 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 05:22:56.493300 23692 rpc_server.cc:307] RPC server started. Bound to: 127.31.100.62:40577
I20260812 05:22:56.493338 23815 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.100.62:40577 every 8 connection(s)
I20260812 05:22:56.493719 23692 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786512017613366-32144-0/minicluster-data/master-0/data/info.pb
I20260812 05:22:56.495306 32144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0waFvT/build/release/bin/kudu as pid 23692
I20260812 05:22:56.495332 23816 sys_catalog.cc:263] Verifying existing consensus state
I20260812 05:22:56.495571 32144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0waFvT/build/release/bin/kudu
/tmp/dist-test-task0waFvT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786512017613366-32144-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786512017613366-32144-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786512017613366-32144-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786512017613366-32144-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.100.1:44501
--local_ip_for_outbound_sockets=127.31.100.1
--tserver_master_addrs=127.31.100.62:40577
--webserver_port=43341
--webserver_interface=127.31.100.1
--keytab_file=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.1.keytab
--principal=kudu/127.31.100.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.100.20:37413
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786512017613366-32144-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786512017613366-32144-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786512017613366-32144-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 05:22:56.496053 23816 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c866d37f66fb4b64bae37566f2baad8e: Bootstrap starting.
I20260812 05:22:56.504356 23816 log.cc:824] T 00000000000000000000000000000000 P c866d37f66fb4b64bae37566f2baad8e: Log is configured to *not* fsync() on all Append() calls
I20260812 05:22:56.506381 23816 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c866d37f66fb4b64bae37566f2baad8e: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 05:22:56.506666 23816 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c866d37f66fb4b64bae37566f2baad8e: Bootstrap complete.
I20260812 05:22:56.508765 23816 raft_consensus.cc:348] T 00000000000000000000000000000000 P c866d37f66fb4b64bae37566f2baad8e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c866d37f66fb4b64bae37566f2baad8e" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 40577 } }
I20260812 05:22:56.508982 23816 raft_consensus.cc:729] T 00000000000000000000000000000000 P c866d37f66fb4b64bae37566f2baad8e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c866d37f66fb4b64bae37566f2baad8e, State: Initialized, Role: FOLLOWER
I20260812 05:22:56.509132 23816 consensus_queue.cc:260] T 00000000000000000000000000000000 P c866d37f66fb4b64bae37566f2baad8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c866d37f66fb4b64bae37566f2baad8e" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 40577 } }
I20260812 05:22:56.509195 23816 raft_consensus.cc:388] T 00000000000000000000000000000000 P c866d37f66fb4b64bae37566f2baad8e [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 05:22:56.509243 23816 raft_consensus.cc:482] T 00000000000000000000000000000000 P c866d37f66fb4b64bae37566f2baad8e [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 05:22:56.509290 23816 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c866d37f66fb4b64bae37566f2baad8e [term 1 FOLLOWER]: Advancing to term 2
I20260812 05:22:56.510286 23816 raft_consensus.cc:504] T 00000000000000000000000000000000 P c866d37f66fb4b64bae37566f2baad8e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c866d37f66fb4b64bae37566f2baad8e" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 40577 } }
I20260812 05:22:56.510370 23816 leader_election.cc:302] T 00000000000000000000000000000000 P c866d37f66fb4b64bae37566f2baad8e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c866d37f66fb4b64bae37566f2baad8e; no voters:
I20260812 05:22:56.510527 23816 leader_election.cc:288] T 00000000000000000000000000000000 P c866d37f66fb4b64bae37566f2baad8e [CANDIDATE]: Term 2 election: Requested vote from peers
I20260812 05:22:56.510600 23823 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c866d37f66fb4b64bae37566f2baad8e [term 2 FOLLOWER]: Leader election won for term 2
I20260812 05:22:56.510751 23823 raft_consensus.cc:686] T 00000000000000000000000000000000 P c866d37f66fb4b64bae37566f2baad8e [term 2 LEADER]: Becoming Leader. State: Replica: c866d37f66fb4b64bae37566f2baad8e, State: Running, Role: LEADER
I20260812 05:22:56.510759 23816 sys_catalog.cc:564] T 00000000000000000000000000000000 P c866d37f66fb4b64bae37566f2baad8e [sys.catalog]: configured and running, proceeding with master startup.
I20260812 05:22:56.511466 23823 consensus_queue.cc:237] T 00000000000000000000000000000000 P c866d37f66fb4b64bae37566f2baad8e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c866d37f66fb4b64bae37566f2baad8e" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 40577 } }
I20260812 05:22:56.512001 23824 sys_catalog.cc:455] T 00000000000000000000000000000000 P c866d37f66fb4b64bae37566f2baad8e [sys.catalog]: SysCatalogTable state changed. Reason: New leader c866d37f66fb4b64bae37566f2baad8e. Latest consensus state: current_term: 2 leader_uuid: "c866d37f66fb4b64bae37566f2baad8e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c866d37f66fb4b64bae37566f2baad8e" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 40577 } } }
I20260812 05:22:56.512085 23824 sys_catalog.cc:458] T 00000000000000000000000000000000 P c866d37f66fb4b64bae37566f2baad8e [sys.catalog]: This master's current role is: LEADER
I20260812 05:22:56.512353 23825 sys_catalog.cc:455] T 00000000000000000000000000000000 P c866d37f66fb4b64bae37566f2baad8e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "c866d37f66fb4b64bae37566f2baad8e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c866d37f66fb4b64bae37566f2baad8e" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 40577 } } }
I20260812 05:22:56.512421 23825 sys_catalog.cc:458] T 00000000000000000000000000000000 P c866d37f66fb4b64bae37566f2baad8e [sys.catalog]: This master's current role is: LEADER
I20260812 05:22:56.512459 23857 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 05:22:56.513221 23857 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 05:22:56.513322 23857 catalog_manager.cc:1259] Loaded cluster ID: ece460ce7ad94ebfacddc2bb26fd457b
I20260812 05:22:56.513353 23857 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 05:22:56.513610 23857 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260812 05:22:56.518210 23857 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260812 05:22:56.520498 23857 catalog_manager.cc:1504] Loading token signing keys...
I20260812 05:22:56.520705 23857 catalog_manager.cc:5879] T 00000000000000000000000000000000 P c866d37f66fb4b64bae37566f2baad8e: Loaded TSK: 0
I20260812 05:22:56.520915 23857 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260812 05:22:56.578204 23819 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 05:22:56.578444 23819 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 05:22:56.578476 23819 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 05:22:56.579596 23819 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 05:22:56.579634 23819 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 05:22:56.579667 23819 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.100.1
I20260812 05:22:56.581105 23819 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.100.20:37413
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786512017613366-32144-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786512017613366-32144-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.1.keytab
--principal=kudu/127.31.100.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.100.1:44501
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786512017613366-32144-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.100.1
--webserver_port=43341
--tserver_master_addrs=127.31.100.62:40577
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23819
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.100.1
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786512017613366-32144-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 04:20:34 UTC on ccycloud.quasar-phqdqp.root.comops.site
build id 4581871
I20260812 05:22:56.581323 23819 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 05:22:56.581559 23819 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 05:22:56.586254 23879 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 05:22:56.586288 23874 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 05:22:56.586246 23875 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 05:22:56.587241 23819 server_base.cc:1034] running on GCE node
I20260812 05:22:56.587438 23819 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 05:22:56.587658 23819 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 05:22:56.588795 23819 hybrid_clock.cc:648] HybridClock initialized: now 1786512176588782 us; error 31 us; skew 500 ppm
Aug 12 05:22:56 dist-test-slave-42z9 krb5kdc[22682](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512176, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.100.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 05:22:56.591269 23819 init.cc:377] Logged in from keytab as kudu/127.31.100.1@KRBTEST.COM (short username kudu)
I20260812 05:22:56.591698 23819 webserver.cc:460] Webserver started at http://127.31.100.1:43341/ using document root <none> and password file <none>
I20260812 05:22:56.591897 23819 fs_manager.cc:359] Metadata directory not provided
I20260812 05:22:56.591957 23819 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 05:22:56.593211 23819 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260812 05:22:56.593873 23890 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 05:22:56.594033 23819 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260812 05:22:56.594118 23819 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786512017613366-32144-0/minicluster-data/ts-0/data,/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786512017613366-32144-0/minicluster-data/ts-0/wal
uuid: "e183f30ebe8b419f8b2fdec556634e45"
format_stamp: "Formatted at 2026-08-12 05:22:55 on dist-test-slave-42z9"
server_key: "79fc4ee8ece2e5b458a42b62eebdf8e6"
server_key_iv: "ce475b6e38a19f92ff73166e94d397a9"
server_key_version: "encryptionkey@0"
I20260812 05:22:56.594431 23819 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786512017613366-32144-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786512017613366-32144-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786512017613366-32144-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 05:22:56.606554 23819 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 05:22:56.608433 23819 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 05:22:56.608557 23819 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 05:22:56.608764 23819 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 05:22:56.609189 23897 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260812 05:22:56.610567 23819 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260812 05:22:56.610630 23819 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260812 05:22:56.610668 23819 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260812 05:22:56.611608 23819 ts_tablet_manager.cc:610] Registered 1 tablets
I20260812 05:22:56.611657 23819 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.001s sys 0.000s
I20260812 05:22:56.611699 23897 tablet_bootstrap.cc:492] T 1c4503a6bf6b45fa9f2e51edc3e83ce1 P e183f30ebe8b419f8b2fdec556634e45: Bootstrap starting.
I20260812 05:22:56.619040 23819 rpc_server.cc:307] RPC server started. Bound to: 127.31.100.1:44501
I20260812 05:22:56.619168 24089 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.100.1:44501 every 8 connection(s)
I20260812 05:22:56.619486 23819 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786512017613366-32144-0/minicluster-data/ts-0/data/info.pb
I20260812 05:22:56.621060 32144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0waFvT/build/release/bin/kudu as pid 23819
I20260812 05:22:56.621369 32144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0waFvT/build/release/bin/kudu
/tmp/dist-test-task0waFvT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786512017613366-32144-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786512017613366-32144-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786512017613366-32144-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786512017613366-32144-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.100.2:40255
--local_ip_for_outbound_sockets=127.31.100.2
--tserver_master_addrs=127.31.100.62:40577
--webserver_port=42543
--webserver_interface=127.31.100.2
--keytab_file=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.2.keytab
--principal=kudu/127.31.100.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.100.20:37413
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786512017613366-32144-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786512017613366-32144-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786512017613366-32144-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 05:22:56.621539 23897 log.cc:824] T 1c4503a6bf6b45fa9f2e51edc3e83ce1 P e183f30ebe8b419f8b2fdec556634e45: Log is configured to *not* fsync() on all Append() calls
I20260812 05:22:56.623970 23897 tablet_bootstrap.cc:492] T 1c4503a6bf6b45fa9f2e51edc3e83ce1 P e183f30ebe8b419f8b2fdec556634e45: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 05:22:56.624217 23897 tablet_bootstrap.cc:492] T 1c4503a6bf6b45fa9f2e51edc3e83ce1 P e183f30ebe8b419f8b2fdec556634e45: Bootstrap complete.
I20260812 05:22:56.625211 23897 ts_tablet_manager.cc:1397] T 1c4503a6bf6b45fa9f2e51edc3e83ce1 P e183f30ebe8b419f8b2fdec556634e45: Time spent bootstrapping tablet: real 0.014s user 0.008s sys 0.003s
I20260812 05:22:56.626339 23897 raft_consensus.cc:348] T 1c4503a6bf6b45fa9f2e51edc3e83ce1 P e183f30ebe8b419f8b2fdec556634e45 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "323e33cb599a49dc8c4ad5ca105d6634" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 40255 } } peers { permanent_uuid: "e183f30ebe8b419f8b2fdec556634e45" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 44501 } } peers { permanent_uuid: "4d517dee1abe4f2e8070c046252ea7be" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 44797 } }
I20260812 05:22:56.626663 23897 raft_consensus.cc:729] T 1c4503a6bf6b45fa9f2e51edc3e83ce1 P e183f30ebe8b419f8b2fdec556634e45 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e183f30ebe8b419f8b2fdec556634e45, State: Initialized, Role: FOLLOWER
I20260812 05:22:56.626801 23897 consensus_queue.cc:260] T 1c4503a6bf6b45fa9f2e51edc3e83ce1 P e183f30ebe8b419f8b2fdec556634e45 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "323e33cb599a49dc8c4ad5ca105d6634" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 40255 } } peers { permanent_uuid: "e183f30ebe8b419f8b2fdec556634e45" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 44501 } } peers { permanent_uuid: "4d517dee1abe4f2e8070c046252ea7be" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 44797 } }
I20260812 05:22:56.627090 23897 ts_tablet_manager.cc:1428] T 1c4503a6bf6b45fa9f2e51edc3e83ce1 P e183f30ebe8b419f8b2fdec556634e45: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
Aug 12 05:22:56 dist-test-slave-42z9 krb5kdc[22682](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512176, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.100.1@KRBTEST.COM for kudu/127.31.100.62@KRBTEST.COM
I20260812 05:22:56.632810 24093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:22:56.620167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.62:40577 (local address 127.31.100.1:53141)
0812 05:22:56.620372 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0812 05:22:56.620420 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0812 05:22:56.620999 (+ 579us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:22:56.622216 (+ 1217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:22:56.622220 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:22:56.622573 (+ 353us) client_negotiation.cc:189] Negotiated authn=SASL
0812 05:22:56.622830 (+ 257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:22:56.622837 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:22:56.624561 (+ 1724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:22:56.624562 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:22:56.625766 (+ 1204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:22:56.625769 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:22:56.626538 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:22:56.626540 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:22:56.626674 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:22:56.627211 (+ 537us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 05:22:56.627221 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0812 05:22:56.628950 (+ 1729us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 05:22:56.631124 (+ 2174us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 05:22:56.631130 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 05:22:56.631132 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 05:22:56.631403 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 05:22:56.631665 (+ 262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 05:22:56.631666 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 05:22:56.631666 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 05:22:56.631789 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 05:22:56.632212 (+ 423us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 05:22:56.632218 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 05:22:56.632357 (+ 139us) client_negotiation.cc:769] Sending connection context
0812 05:22:56.632453 (+ 96us) client_negotiation.cc:240] Negotiation successful
0812 05:22:56.632608 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":109,"threads_started":1}
I20260812 05:22:56.633327 24096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:22:56.620261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.100.1:53141 (local address 127.31.100.62:40577)
0812 05:22:56.621789 (+ 1528us) server_negotiation.cc:206] Beginning negotiation
0812 05:22:56.621797 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 05:22:56.621831 (+ 34us) server_negotiation.cc:407] Connection header received
0812 05:22:56.621955 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:22:56.621962 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:22:56.622017 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:22:56.622169 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:22:56.622970 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:22:56.624447 (+ 1477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:22:56.625854 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:22:56.626434 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:22:56.626541 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:22:56.629068 (+ 2527us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 05:22:56.629071 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 05:22:56.629072 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 05:22:56.629083 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0812 05:22:56.631035 (+ 1952us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:22:56.631485 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:22:56.631488 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:22:56.631488 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:22:56.631599 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:22:56.631850 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:22:56.631851 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:22:56.631851 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:22:56.632117 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 05:22:56.632185 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0812 05:22:56.632996 (+ 811us) server_negotiation.cc:299] Negotiation successful
0812 05:22:56.633125 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1479,"thread_start_us":70,"threads_started":1}
I20260812 05:22:56.634079 24090 heartbeater.cc:347] Connected to a master server at 127.31.100.62:40577
I20260812 05:22:56.634186 24090 heartbeater.cc:464] Registering TS with master...
I20260812 05:22:56.634377 24090 heartbeater.cc:511] Master 127.31.100.62:40577 requested a full tablet report, sending...
I20260812 05:22:56.635183 23747 ts_manager.cc:194] Registered new tserver with Master: e183f30ebe8b419f8b2fdec556634e45 (127.31.100.1:44501)
I20260812 05:22:56.636369 23747 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.100.1@KRBTEST.COM'} at 127.31.100.1:53141
I20260812 05:22:56.637331 24090 heartbeater.cc:503] Master 127.31.100.62:40577 was elected leader, sending a full tablet report...
Aug 12 05:22:56 dist-test-slave-42z9 krb5kdc[22682](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512176, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.100.62@KRBTEST.COM for kudu/127.31.100.1@KRBTEST.COM
I20260812 05:22:56.645402 24113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:22:56.635812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.1:44501 (local address 127.0.0.1:54860)
0812 05:22:56.636231 (+ 419us) negotiation.cc:107] Waiting for socket to connect
0812 05:22:56.636269 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0812 05:22:56.636332 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:22:56.636843 (+ 511us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:22:56.636849 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:22:56.637175 (+ 326us) client_negotiation.cc:189] Negotiated authn=SASL
0812 05:22:56.637229 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:22:56.637231 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:22:56.638794 (+ 1563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:22:56.638795 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:22:56.639687 (+ 892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:22:56.639690 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:22:56.640288 (+ 598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:22:56.640288 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:22:56.640361 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:22:56.640808 (+ 447us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 05:22:56.640813 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0812 05:22:56.642401 (+ 1588us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 05:22:56.644312 (+ 1911us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 05:22:56.644319 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 05:22:56.644321 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 05:22:56.644548 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 05:22:56.644737 (+ 189us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 05:22:56.644737 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 05:22:56.644738 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 05:22:56.644799 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 05:22:56.645075 (+ 276us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 05:22:56.645081 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 05:22:56.645130 (+ 49us) client_negotiation.cc:769] Sending connection context
0812 05:22:56.645207 (+ 77us) client_negotiation.cc:240] Negotiation successful
0812 05:22:56.645313 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":388,"thread_start_us":82,"threads_started":1}
I20260812 05:22:56.645457 24115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:22:56.636066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54860 (local address 127.31.100.1:44501)
0812 05:22:56.636452 (+ 386us) server_negotiation.cc:206] Beginning negotiation
0812 05:22:56.636456 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 05:22:56.636465 (+ 9us) server_negotiation.cc:407] Connection header received
0812 05:22:56.636496 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:22:56.636498 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:22:56.636520 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:22:56.636584 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:22:56.637318 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:22:56.638708 (+ 1390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:22:56.639766 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:22:56.640221 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:22:56.640285 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:22:56.642485 (+ 2200us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 05:22:56.642493 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 05:22:56.642493 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 05:22:56.642500 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 05:22:56.644226 (+ 1726us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:22:56.644621 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:22:56.644624 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:22:56.644624 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:22:56.644682 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:22:56.644848 (+ 166us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:22:56.644849 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:22:56.644849 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:22:56.645005 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 05:22:56.645094 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0812 05:22:56.645252 (+ 158us) server_negotiation.cc:299] Negotiation successful
0812 05:22:56.645345 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":355,"thread_start_us":77,"threads_started":1}
I20260812 05:22:56.646210 23972 tablet_service.cc:1514] Processing DeleteTablet for tablet 1c4503a6bf6b45fa9f2e51edc3e83ce1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 05:22:56 UTC) from {username='kudu', principal='kudu/127.31.100.62@KRBTEST.COM'} at 127.0.0.1:54860
I20260812 05:22:56.646828 24117 tablet_replica.cc:330] stopping tablet replica
I20260812 05:22:56.646929 24117 raft_consensus.cc:2229] T 1c4503a6bf6b45fa9f2e51edc3e83ce1 P e183f30ebe8b419f8b2fdec556634e45 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 05:22:56.646997 24117 raft_consensus.cc:2258] T 1c4503a6bf6b45fa9f2e51edc3e83ce1 P e183f30ebe8b419f8b2fdec556634e45 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 05:22:56.647285 24117 ts_tablet_manager.cc:1905] T 1c4503a6bf6b45fa9f2e51edc3e83ce1 P e183f30ebe8b419f8b2fdec556634e45: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 05:22:56.649142 24117 ts_tablet_manager.cc:1918] T 1c4503a6bf6b45fa9f2e51edc3e83ce1 P e183f30ebe8b419f8b2fdec556634e45: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 05:22:56.649207 24117 log.cc:1196] T 1c4503a6bf6b45fa9f2e51edc3e83ce1 P e183f30ebe8b419f8b2fdec556634e45: Deleting WAL directory at /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786512017613366-32144-0/minicluster-data/ts-0/wal/wals/1c4503a6bf6b45fa9f2e51edc3e83ce1
I20260812 05:22:56.649400 24117 ts_tablet_manager.cc:1939] T 1c4503a6bf6b45fa9f2e51edc3e83ce1 P e183f30ebe8b419f8b2fdec556634e45: Deleting consensus metadata
I20260812 05:22:56.649871 23724 catalog_manager.cc:4841] TS e183f30ebe8b419f8b2fdec556634e45 (127.31.100.1:44501): tablet 1c4503a6bf6b45fa9f2e51edc3e83ce1 (table test-table [id=acedb8b609074c53b6f443b82e7a6a8d]) successfully deleted
W20260812 05:22:56.711951 24104 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 05:22:56.712141 24104 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 05:22:56.712163 24104 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 05:22:56.713243 24104 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 05:22:56.713276 24104 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 05:22:56.713300 24104 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.100.2
I20260812 05:22:56.714708 24104 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.100.20:37413
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786512017613366-32144-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786512017613366-32144-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.2.keytab
--principal=kudu/127.31.100.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.100.2:40255
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786512017613366-32144-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.100.2
--webserver_port=42543
--tserver_master_addrs=127.31.100.62:40577
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24104
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.100.2
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786512017613366-32144-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 04:20:34 UTC on ccycloud.quasar-phqdqp.root.comops.site
build id 4581871
I20260812 05:22:56.714901 24104 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 05:22:56.715075 24104 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 05:22:56.719719 24125 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 05:22:56.719734 24129 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 05:22:56.719764 24126 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 05:22:56.719934 24104 server_base.cc:1034] running on GCE node
I20260812 05:22:56.720160 24104 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 05:22:56.720356 24104 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 05:22:56.721508 24104 hybrid_clock.cc:648] HybridClock initialized: now 1786512176721491 us; error 33 us; skew 500 ppm
Aug 12 05:22:56 dist-test-slave-42z9 krb5kdc[22682](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512176, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.100.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 05:22:56.723758 24104 init.cc:377] Logged in from keytab as kudu/127.31.100.2@KRBTEST.COM (short username kudu)
I20260812 05:22:56.724129 24104 webserver.cc:460] Webserver started at http://127.31.100.2:42543/ using document root <none> and password file <none>
I20260812 05:22:56.724310 24104 fs_manager.cc:359] Metadata directory not provided
I20260812 05:22:56.724361 24104 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 05:22:56.725579 24104 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260812 05:22:56.726078 24145 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 05:22:56.726225 24104 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260812 05:22:56.726305 24104 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786512017613366-32144-0/minicluster-data/ts-1/data,/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786512017613366-32144-0/minicluster-data/ts-1/wal
uuid: "323e33cb599a49dc8c4ad5ca105d6634"
format_stamp: "Formatted at 2026-08-12 05:22:55 on dist-test-slave-42z9"
server_key: "ec87d2ef04a1a1fe9b2bd7e62ab79afe"
server_key_iv: "673423269407aaced0bfb9334e1c7d54"
server_key_version: "encryptionkey@0"
I20260812 05:22:56.726589 24104 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786512017613366-32144-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786512017613366-32144-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786512017613366-32144-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 05:22:56.743458 24104 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 05:22:56.745152 24104 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 05:22:56.745255 24104 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 05:22:56.745414 24104 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 05:22:56.745687 24104 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 05:22:56.745720 24104 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 05:22:56.745739 24104 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 05:22:56.745754 24104 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 05:22:56.751130 24104 rpc_server.cc:307] RPC server started. Bound to: 127.31.100.2:40255
I20260812 05:22:56.751194 24370 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.100.2:40255 every 8 connection(s)
I20260812 05:22:56.751473 24104 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786512017613366-32144-0/minicluster-data/ts-1/data/info.pb
I20260812 05:22:56.752233 32144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0waFvT/build/release/bin/kudu as pid 24104
I20260812 05:22:56.752481 32144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0waFvT/build/release/bin/kudu
/tmp/dist-test-task0waFvT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786512017613366-32144-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786512017613366-32144-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786512017613366-32144-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786512017613366-32144-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.100.3:44797
--local_ip_for_outbound_sockets=127.31.100.3
--tserver_master_addrs=127.31.100.62:40577
--webserver_port=38583
--webserver_interface=127.31.100.3
--keytab_file=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.3.keytab
--principal=kudu/127.31.100.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.100.20:37413
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786512017613366-32144-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786512017613366-32144-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786512017613366-32144-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 12 05:22:56 dist-test-slave-42z9 krb5kdc[22682](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512176, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.100.2@KRBTEST.COM for kudu/127.31.100.62@KRBTEST.COM
I20260812 05:22:56.761058 24096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:22:56.752138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.100.2:35519 (local address 127.31.100.62:40577)
0812 05:22:56.752203 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0812 05:22:56.752205 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 05:22:56.752470 (+ 265us) server_negotiation.cc:407] Connection header received
0812 05:22:56.752995 (+ 525us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:22:56.752997 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:22:56.753012 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:22:56.753068 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:22:56.754011 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:22:56.754527 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:22:56.755829 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:22:56.756035 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:22:56.756092 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:22:56.758403 (+ 2311us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 05:22:56.758405 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 05:22:56.758405 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 05:22:56.758412 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 05:22:56.759637 (+ 1225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:22:56.760094 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:22:56.760095 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:22:56.760096 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:22:56.760140 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:22:56.760417 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:22:56.760418 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:22:56.760419 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:22:56.760538 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 05:22:56.760602 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0812 05:22:56.760908 (+ 306us) server_negotiation.cc:299] Negotiation successful
0812 05:22:56.760991 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260812 05:22:56.761273 24377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:22:56.752129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.62:40577 (local address 127.31.100.2:35519)
0812 05:22:56.752353 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0812 05:22:56.752389 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0812 05:22:56.752896 (+ 507us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:22:56.753134 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:22:56.753137 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:22:56.753702 (+ 565us) client_negotiation.cc:189] Negotiated authn=SASL
0812 05:22:56.753896 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:22:56.753903 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:22:56.754617 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:22:56.754617 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:22:56.755739 (+ 1122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:22:56.755743 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:22:56.756119 (+ 376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:22:56.756120 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:22:56.756248 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:22:56.756691 (+ 443us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 05:22:56.756702 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0812 05:22:56.758313 (+ 1611us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 05:22:56.759715 (+ 1402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 05:22:56.759723 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 05:22:56.759725 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 05:22:56.760028 (+ 303us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 05:22:56.760194 (+ 166us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 05:22:56.760195 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 05:22:56.760195 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 05:22:56.760328 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 05:22:56.760608 (+ 280us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 05:22:56.760614 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 05:22:56.760770 (+ 156us) client_negotiation.cc:769] Sending connection context
0812 05:22:56.760892 (+ 122us) client_negotiation.cc:240] Negotiation successful
0812 05:22:56.761056 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":80,"threads_started":1}
I20260812 05:22:56.761785 24372 heartbeater.cc:347] Connected to a master server at 127.31.100.62:40577
I20260812 05:22:56.761901 24372 heartbeater.cc:464] Registering TS with master...
I20260812 05:22:56.762054 24372 heartbeater.cc:511] Master 127.31.100.62:40577 requested a full tablet report, sending...
I20260812 05:22:56.762408 23747 ts_manager.cc:194] Registered new tserver with Master: 323e33cb599a49dc8c4ad5ca105d6634 (127.31.100.2:40255)
I20260812 05:22:56.762794 23747 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.100.2@KRBTEST.COM'} at 127.31.100.2:35519
W20260812 05:22:56.834457 24380 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 05:22:56.834642 24380 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 05:22:56.834663 24380 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 05:22:56.835733 24380 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 05:22:56.835770 24380 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 05:22:56.835794 24380 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.100.3
I20260812 05:22:56.837188 24380 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.100.20:37413
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786512017613366-32144-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786512017613366-32144-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.3.keytab
--principal=kudu/127.31.100.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.100.3:44797
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786512017613366-32144-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.100.3
--webserver_port=38583
--tserver_master_addrs=127.31.100.62:40577
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24380
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.100.3
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786512017613366-32144-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 04:20:34 UTC on ccycloud.quasar-phqdqp.root.comops.site
build id 4581871
I20260812 05:22:56.837402 24380 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 05:22:56.837615 24380 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 05:22:56.842236 24394 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 05:22:56.842213 24391 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 05:22:56.842221 24397 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 05:22:56.842428 24380 server_base.cc:1034] running on GCE node
I20260812 05:22:56.842628 24380 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 05:22:56.842861 24380 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 05:22:56.844012 24380 hybrid_clock.cc:648] HybridClock initialized: now 1786512176843992 us; error 31 us; skew 500 ppm
Aug 12 05:22:56 dist-test-slave-42z9 krb5kdc[22682](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512176, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.100.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 05:22:56.846376 24380 init.cc:377] Logged in from keytab as kudu/127.31.100.3@KRBTEST.COM (short username kudu)
I20260812 05:22:56.846792 24380 webserver.cc:460] Webserver started at http://127.31.100.3:38583/ using document root <none> and password file <none>
I20260812 05:22:56.846992 24380 fs_manager.cc:359] Metadata directory not provided
I20260812 05:22:56.847055 24380 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 05:22:56.848242 24380 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260812 05:22:56.848788 24409 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 05:22:56.848949 24380 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260812 05:22:56.849041 24380 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786512017613366-32144-0/minicluster-data/ts-2/data,/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786512017613366-32144-0/minicluster-data/ts-2/wal
uuid: "4d517dee1abe4f2e8070c046252ea7be"
format_stamp: "Formatted at 2026-08-12 05:22:56 on dist-test-slave-42z9"
server_key: "3b6ae4bd9db96d6907c558c9f73f404b"
server_key_iv: "44679731b0da20a67ac4afea59c62f29"
server_key_version: "encryptionkey@0"
I20260812 05:22:56.849325 24380 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786512017613366-32144-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786512017613366-32144-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786512017613366-32144-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 05:22:56.872082 24380 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 05:22:56.874033 24380 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 05:22:56.874169 24380 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 05:22:56.874366 24380 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 05:22:56.874646 24380 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 05:22:56.874678 24380 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 05:22:56.874706 24380 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 05:22:56.874723 24380 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 05:22:56.880201 24380 rpc_server.cc:307] RPC server started. Bound to: 127.31.100.3:44797
I20260812 05:22:56.880254 24616 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.100.3:44797 every 8 connection(s)
I20260812 05:22:56.880556 24380 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786512017613366-32144-0/minicluster-data/ts-2/data/info.pb
Aug 12 05:22:56 dist-test-slave-42z9 krb5kdc[22682](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512176, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.100.3@KRBTEST.COM for kudu/127.31.100.62@KRBTEST.COM
I20260812 05:22:56.888386 32144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0waFvT/build/release/bin/kudu as pid 24380
I20260812 05:22:56.888409 24096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:22:56.881166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.100.3:39179 (local address 127.31.100.62:40577)
0812 05:22:56.881244 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0812 05:22:56.881246 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 05:22:56.881470 (+ 224us) server_negotiation.cc:407] Connection header received
0812 05:22:56.881902 (+ 432us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:22:56.881903 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:22:56.881913 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:22:56.881975 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:22:56.882800 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:22:56.883278 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:22:56.884183 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:22:56.884375 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:22:56.884438 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:22:56.886252 (+ 1814us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 05:22:56.886253 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 05:22:56.886254 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 05:22:56.886260 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0812 05:22:56.887327 (+ 1067us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:22:56.887676 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:22:56.887677 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:22:56.887677 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:22:56.887734 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:22:56.887935 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:22:56.887935 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:22:56.887935 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:22:56.888037 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 05:22:56.888082 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0812 05:22:56.888286 (+ 204us) server_negotiation.cc:299] Negotiation successful
0812 05:22:56.888348 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260812 05:22:56.888648 24621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:22:56.881160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.62:40577 (local address 127.31.100.3:39179)
0812 05:22:56.881312 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0812 05:22:56.881344 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0812 05:22:56.881822 (+ 478us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:22:56.882026 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:22:56.882029 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:22:56.882546 (+ 517us) client_negotiation.cc:189] Negotiated authn=SASL
0812 05:22:56.882721 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:22:56.882725 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:22:56.883341 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:22:56.883341 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:22:56.884102 (+ 761us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:22:56.884103 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:22:56.884447 (+ 344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:22:56.884448 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:22:56.884546 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:22:56.884907 (+ 361us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 05:22:56.884914 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0812 05:22:56.886172 (+ 1258us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 05:22:56.887406 (+ 1234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 05:22:56.887411 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 05:22:56.887413 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 05:22:56.887611 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 05:22:56.887792 (+ 181us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 05:22:56.887792 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 05:22:56.887793 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 05:22:56.887885 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 05:22:56.888080 (+ 195us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 05:22:56.888082 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 05:22:56.888182 (+ 100us) client_negotiation.cc:769] Sending connection context
0812 05:22:56.888275 (+ 93us) client_negotiation.cc:240] Negotiation successful
0812 05:22:56.888460 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":84,"threads_started":1}
I20260812 05:22:56.889094 24617 heartbeater.cc:347] Connected to a master server at 127.31.100.62:40577
I20260812 05:22:56.889185 24617 heartbeater.cc:464] Registering TS with master...
I20260812 05:22:56.889346 24617 heartbeater.cc:511] Master 127.31.100.62:40577 requested a full tablet report, sending...
I20260812 05:22:56.889719 23747 ts_manager.cc:194] Registered new tserver with Master: 4d517dee1abe4f2e8070c046252ea7be (127.31.100.3:44797)
I20260812 05:22:56.890098 23747 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.100.3@KRBTEST.COM'} at 127.31.100.3:39179
I20260812 05:22:56.893942 24096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:22:56.888750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59854 (local address 127.31.100.62:40577)
0812 05:22:56.888785 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0812 05:22:56.888787 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 05:22:56.888921 (+ 134us) server_negotiation.cc:407] Connection header received
0812 05:22:56.888989 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:22:56.888990 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:22:56.889000 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:22:56.889047 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:22:56.889677 (+ 630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:22:56.890185 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:22:56.890907 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:22:56.891080 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:22:56.891134 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:22:56.892154 (+ 1020us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 05:22:56.892155 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 05:22:56.892155 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 05:22:56.892161 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0812 05:22:56.893148 (+ 987us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:22:56.893427 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:22:56.893427 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:22:56.893427 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:22:56.893496 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:22:56.893633 (+ 137us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:22:56.893633 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:22:56.893633 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:22:56.893713 (+ 80us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 05:22:56.893810 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0812 05:22:56.893838 (+ 28us) server_negotiation.cc:299] Negotiation successful
0812 05:22:56.893888 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260812 05:22:56.894551 32144 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 05:22:56.896862 24096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:22:56.894854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59856 (local address 127.31.100.62:40577)
0812 05:22:56.894890 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0812 05:22:56.894891 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0812 05:22:56.894992 (+ 101us) server_negotiation.cc:407] Connection header received
0812 05:22:56.895035 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:22:56.895036 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:22:56.895062 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:22:56.895122 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 05:22:56.895190 (+ 68us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:22:56.895604 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:22:56.896307 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:22:56.896502 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:22:56.896546 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:22:56.896667 (+ 121us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 05:22:56.896746 (+ 79us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 05:22:56.896791 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0812 05:22:56.896802 (+ 11us) server_negotiation.cc:299] Negotiation successful
0812 05:22:56.896814 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260812 05:22:56.897136 23747 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:59856:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 05:22:56.898733 23747 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 05:22:56.900712 23972 tablet_service.cc:1467] Processing CreateTablet for tablet f1605964bd9443e7a6fa66525884030d (DEFAULT_TABLE table=test-table [id=3e8453ed852a4078b5b017c8146f7b0f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 05:22:56.900921 23972 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f1605964bd9443e7a6fa66525884030d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 05:22:56.902532 23897 tablet_bootstrap.cc:492] T f1605964bd9443e7a6fa66525884030d P e183f30ebe8b419f8b2fdec556634e45: Bootstrap starting.
I20260812 05:22:56.902908 23897 tablet_bootstrap.cc:654] T f1605964bd9443e7a6fa66525884030d P e183f30ebe8b419f8b2fdec556634e45: Neither blocks nor log segments found. Creating new log.
I20260812 05:22:56.903383 23897 tablet_bootstrap.cc:492] T f1605964bd9443e7a6fa66525884030d P e183f30ebe8b419f8b2fdec556634e45: No bootstrap required, opened a new log
I20260812 05:22:56.903436 23897 ts_tablet_manager.cc:1397] T f1605964bd9443e7a6fa66525884030d P e183f30ebe8b419f8b2fdec556634e45: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260812 05:22:56.903775 23897 raft_consensus.cc:348] T f1605964bd9443e7a6fa66525884030d P e183f30ebe8b419f8b2fdec556634e45 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "323e33cb599a49dc8c4ad5ca105d6634" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 40255 } } peers { permanent_uuid: "4d517dee1abe4f2e8070c046252ea7be" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 44797 } } peers { permanent_uuid: "e183f30ebe8b419f8b2fdec556634e45" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 44501 } }
I20260812 05:22:56.903861 23897 raft_consensus.cc:374] T f1605964bd9443e7a6fa66525884030d P e183f30ebe8b419f8b2fdec556634e45 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 05:22:56.903925 23897 raft_consensus.cc:729] T f1605964bd9443e7a6fa66525884030d P e183f30ebe8b419f8b2fdec556634e45 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e183f30ebe8b419f8b2fdec556634e45, State: Initialized, Role: FOLLOWER
I20260812 05:22:56.903973 23897 consensus_queue.cc:260] T f1605964bd9443e7a6fa66525884030d P e183f30ebe8b419f8b2fdec556634e45 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "323e33cb599a49dc8c4ad5ca105d6634" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 40255 } } peers { permanent_uuid: "4d517dee1abe4f2e8070c046252ea7be" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 44797 } } peers { permanent_uuid: "e183f30ebe8b419f8b2fdec556634e45" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 44501 } }
I20260812 05:22:56.904076 23897 ts_tablet_manager.cc:1428] T f1605964bd9443e7a6fa66525884030d P e183f30ebe8b419f8b2fdec556634e45: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260812 05:22:56.904168 24625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:22:56.900353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.3:44797 (local address 127.0.0.1:34040)
0812 05:22:56.900505 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0812 05:22:56.900514 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0812 05:22:56.900556 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:22:56.900898 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:22:56.900899 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:22:56.900901 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 05:22:56.900957 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:22:56.900958 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:22:56.901952 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:22:56.901952 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:22:56.903043 (+ 1091us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:22:56.903046 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:22:56.903941 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:22:56.903942 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:22:56.904006 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:22:56.904028 (+ 22us) client_negotiation.cc:769] Sending connection context
0812 05:22:56.904062 (+ 34us) client_negotiation.cc:240] Negotiation successful
0812 05:22:56.904102 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"mutex_wait_us":36,"thread_start_us":80,"threads_started":1}
I20260812 05:22:56.904261 24113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:22:56.900353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.2:40255 (local address 127.0.0.1:32892)
0812 05:22:56.900412 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0812 05:22:56.900422 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0812 05:22:56.900453 (+ 31us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:22:56.900899 (+ 446us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:22:56.900900 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:22:56.900902 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 05:22:56.900942 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:22:56.900944 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:22:56.902186 (+ 1242us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:22:56.902187 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:22:56.903201 (+ 1014us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:22:56.903203 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:22:56.904027 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:22:56.904028 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:22:56.904101 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:22:56.904105 (+ 4us) client_negotiation.cc:769] Sending connection context
0812 05:22:56.904153 (+ 48us) client_negotiation.cc:240] Negotiation successful
0812 05:22:56.904197 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18}
I20260812 05:22:56.904474 24627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:22:56.900434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34040 (local address 127.31.100.3:44797)
0812 05:22:56.900735 (+ 301us) server_negotiation.cc:206] Beginning negotiation
0812 05:22:56.900738 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 05:22:56.900747 (+ 9us) server_negotiation.cc:407] Connection header received
0812 05:22:56.900773 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:22:56.900774 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:22:56.900785 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:22:56.901025 (+ 240us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 05:22:56.901071 (+ 46us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:22:56.901857 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:22:56.903134 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:22:56.903836 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:22:56.904260 (+ 424us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:22:56.904323 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0812 05:22:56.904348 (+ 25us) server_negotiation.cc:299] Negotiation successful
0812 05:22:56.904397 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":276,"thread_start_us":73,"threads_started":1}
I20260812 05:22:56.904739 24631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:22:56.900545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32892 (local address 127.31.100.2:40255)
0812 05:22:56.900713 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0812 05:22:56.900717 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 05:22:56.900726 (+ 9us) server_negotiation.cc:407] Connection header received
0812 05:22:56.900763 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:22:56.900764 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:22:56.900779 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:22:56.901017 (+ 238us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 05:22:56.901082 (+ 65us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:22:56.902085 (+ 1003us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:22:56.903283 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:22:56.903847 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:22:56.903915 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:22:56.903979 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0812 05:22:56.904615 (+ 636us) server_negotiation.cc:299] Negotiation successful
0812 05:22:56.904663 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":93,"threads_started":1}
I20260812 05:22:56.905138 24496 tablet_service.cc:1467] Processing CreateTablet for tablet f1605964bd9443e7a6fa66525884030d (DEFAULT_TABLE table=test-table [id=3e8453ed852a4078b5b017c8146f7b0f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 05:22:56.905349 24232 tablet_service.cc:1467] Processing CreateTablet for tablet f1605964bd9443e7a6fa66525884030d (DEFAULT_TABLE table=test-table [id=3e8453ed852a4078b5b017c8146f7b0f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 05:22:56.905552 24496 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f1605964bd9443e7a6fa66525884030d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 05:22:56.905678 24232 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f1605964bd9443e7a6fa66525884030d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 05:22:56.908892 24635 tablet_bootstrap.cc:492] T f1605964bd9443e7a6fa66525884030d P 4d517dee1abe4f2e8070c046252ea7be: Bootstrap starting.
I20260812 05:22:56.909049 24634 tablet_bootstrap.cc:492] T f1605964bd9443e7a6fa66525884030d P 323e33cb599a49dc8c4ad5ca105d6634: Bootstrap starting.
I20260812 05:22:56.909613 24635 tablet_bootstrap.cc:654] T f1605964bd9443e7a6fa66525884030d P 4d517dee1abe4f2e8070c046252ea7be: Neither blocks nor log segments found. Creating new log.
I20260812 05:22:56.909747 24634 tablet_bootstrap.cc:654] T f1605964bd9443e7a6fa66525884030d P 323e33cb599a49dc8c4ad5ca105d6634: Neither blocks nor log segments found. Creating new log.
I20260812 05:22:56.910187 24635 log.cc:824] T f1605964bd9443e7a6fa66525884030d P 4d517dee1abe4f2e8070c046252ea7be: Log is configured to *not* fsync() on all Append() calls
I20260812 05:22:56.910328 24634 log.cc:824] T f1605964bd9443e7a6fa66525884030d P 323e33cb599a49dc8c4ad5ca105d6634: Log is configured to *not* fsync() on all Append() calls
I20260812 05:22:56.910938 24635 tablet_bootstrap.cc:492] T f1605964bd9443e7a6fa66525884030d P 4d517dee1abe4f2e8070c046252ea7be: No bootstrap required, opened a new log
I20260812 05:22:56.910993 24635 ts_tablet_manager.cc:1397] T f1605964bd9443e7a6fa66525884030d P 4d517dee1abe4f2e8070c046252ea7be: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260812 05:22:56.911002 24634 tablet_bootstrap.cc:492] T f1605964bd9443e7a6fa66525884030d P 323e33cb599a49dc8c4ad5ca105d6634: No bootstrap required, opened a new log
I20260812 05:22:56.911069 24634 ts_tablet_manager.cc:1397] T f1605964bd9443e7a6fa66525884030d P 323e33cb599a49dc8c4ad5ca105d6634: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260812 05:22:56.912339 24635 raft_consensus.cc:348] T f1605964bd9443e7a6fa66525884030d P 4d517dee1abe4f2e8070c046252ea7be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "323e33cb599a49dc8c4ad5ca105d6634" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 40255 } } peers { permanent_uuid: "4d517dee1abe4f2e8070c046252ea7be" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 44797 } } peers { permanent_uuid: "e183f30ebe8b419f8b2fdec556634e45" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 44501 } }
I20260812 05:22:56.912490 24635 raft_consensus.cc:374] T f1605964bd9443e7a6fa66525884030d P 4d517dee1abe4f2e8070c046252ea7be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 05:22:56.912524 24635 raft_consensus.cc:729] T f1605964bd9443e7a6fa66525884030d P 4d517dee1abe4f2e8070c046252ea7be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d517dee1abe4f2e8070c046252ea7be, State: Initialized, Role: FOLLOWER
I20260812 05:22:56.912467 24634 raft_consensus.cc:348] T f1605964bd9443e7a6fa66525884030d P 323e33cb599a49dc8c4ad5ca105d6634 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "323e33cb599a49dc8c4ad5ca105d6634" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 40255 } } peers { permanent_uuid: "4d517dee1abe4f2e8070c046252ea7be" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 44797 } } peers { permanent_uuid: "e183f30ebe8b419f8b2fdec556634e45" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 44501 } }
I20260812 05:22:56.912619 24634 raft_consensus.cc:374] T f1605964bd9443e7a6fa66525884030d P 323e33cb599a49dc8c4ad5ca105d6634 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 05:22:56.912665 24634 raft_consensus.cc:729] T f1605964bd9443e7a6fa66525884030d P 323e33cb599a49dc8c4ad5ca105d6634 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 323e33cb599a49dc8c4ad5ca105d6634, State: Initialized, Role: FOLLOWER
I20260812 05:22:56.912637 24635 consensus_queue.cc:260] T f1605964bd9443e7a6fa66525884030d P 4d517dee1abe4f2e8070c046252ea7be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "323e33cb599a49dc8c4ad5ca105d6634" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 40255 } } peers { permanent_uuid: "4d517dee1abe4f2e8070c046252ea7be" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 44797 } } peers { permanent_uuid: "e183f30ebe8b419f8b2fdec556634e45" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 44501 } }
I20260812 05:22:56.912781 24634 consensus_queue.cc:260] T f1605964bd9443e7a6fa66525884030d P 323e33cb599a49dc8c4ad5ca105d6634 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "323e33cb599a49dc8c4ad5ca105d6634" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 40255 } } peers { permanent_uuid: "4d517dee1abe4f2e8070c046252ea7be" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 44797 } } peers { permanent_uuid: "e183f30ebe8b419f8b2fdec556634e45" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 44501 } }
I20260812 05:22:56.912923 24635 ts_tablet_manager.cc:1428] T f1605964bd9443e7a6fa66525884030d P 4d517dee1abe4f2e8070c046252ea7be: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260812 05:22:56.912981 24617 heartbeater.cc:503] Master 127.31.100.62:40577 was elected leader, sending a full tablet report...
I20260812 05:22:56.913040 24634 ts_tablet_manager.cc:1428] T f1605964bd9443e7a6fa66525884030d P 323e33cb599a49dc8c4ad5ca105d6634: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260812 05:22:56.913174 24372 heartbeater.cc:503] Master 127.31.100.62:40577 was elected leader, sending a full tablet report...
W20260812 05:22:57.002034 24374 tablet.cc:2369] T f1605964bd9443e7a6fa66525884030d P 323e33cb599a49dc8c4ad5ca105d6634: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 05:22:57.128126 24091 tablet.cc:2369] T f1605964bd9443e7a6fa66525884030d P e183f30ebe8b419f8b2fdec556634e45: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 05:22:57.131084 24620 tablet.cc:2369] T f1605964bd9443e7a6fa66525884030d P 4d517dee1abe4f2e8070c046252ea7be: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 05:22:57.132638 24646 raft_consensus.cc:482] T f1605964bd9443e7a6fa66525884030d P 323e33cb599a49dc8c4ad5ca105d6634 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 05:22:57.132746 24646 raft_consensus.cc:504] T f1605964bd9443e7a6fa66525884030d P 323e33cb599a49dc8c4ad5ca105d6634 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "323e33cb599a49dc8c4ad5ca105d6634" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 40255 } } peers { permanent_uuid: "4d517dee1abe4f2e8070c046252ea7be" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 44797 } } peers { permanent_uuid: "e183f30ebe8b419f8b2fdec556634e45" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 44501 } }
I20260812 05:22:57.133123 24646 leader_election.cc:288] T f1605964bd9443e7a6fa66525884030d P 323e33cb599a49dc8c4ad5ca105d6634 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4d517dee1abe4f2e8070c046252ea7be (127.31.100.3:44797), e183f30ebe8b419f8b2fdec556634e45 (127.31.100.1:44501)
I20260812 05:22:57.135972 24627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:22:57.133388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.100.2:38111 (local address 127.31.100.3:44797)
0812 05:22:57.133459 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0812 05:22:57.133461 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 05:22:57.133472 (+ 11us) server_negotiation.cc:407] Connection header received
0812 05:22:57.133574 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:22:57.133575 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:22:57.133585 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:22:57.133640 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 05:22:57.133791 (+ 151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:22:57.134295 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:22:57.135259 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:22:57.135639 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:22:57.135693 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:22:57.135700 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0812 05:22:57.135868 (+ 168us) server_negotiation.cc:299] Negotiation successful
0812 05:22:57.135907 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260812 05:22:57.135967 24377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:22:57.133306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.3:44797 (local address 127.31.100.2:38111)
0812 05:22:57.133376 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0812 05:22:57.133388 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0812 05:22:57.133481 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:22:57.133649 (+ 168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:22:57.133650 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:22:57.133652 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 05:22:57.133729 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:22:57.133731 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:22:57.134371 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:22:57.134372 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:22:57.135167 (+ 795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:22:57.135169 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:22:57.135719 (+ 550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:22:57.135720 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:22:57.135798 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:22:57.135802 (+ 4us) client_negotiation.cc:769] Sending connection context
0812 05:22:57.135851 (+ 49us) client_negotiation.cc:240] Negotiation successful
0812 05:22:57.135892 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52,"mutex_wait_us":1}
I20260812 05:22:57.136041 24115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:22:57.133353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.100.2:58395 (local address 127.31.100.1:44501)
0812 05:22:57.133398 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0812 05:22:57.133401 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 05:22:57.133588 (+ 187us) server_negotiation.cc:407] Connection header received
0812 05:22:57.133652 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:22:57.133653 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:22:57.133677 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:22:57.133745 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 05:22:57.133878 (+ 133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:22:57.134355 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:22:57.135262 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:22:57.135714 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:22:57.135785 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:22:57.135862 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0812 05:22:57.135938 (+ 76us) server_negotiation.cc:299] Negotiation successful
0812 05:22:57.135979 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260812 05:22:57.136089 24648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:22:57.133306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.1:44501 (local address 127.31.100.2:58395)
0812 05:22:57.133508 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0812 05:22:57.133519 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0812 05:22:57.133587 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:22:57.133753 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:22:57.133753 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:22:57.133755 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 05:22:57.133825 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:22:57.133827 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:22:57.134433 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:22:57.134433 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:22:57.135177 (+ 744us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:22:57.135179 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:22:57.135797 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:22:57.135798 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:22:57.135859 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:22:57.135864 (+ 5us) client_negotiation.cc:769] Sending connection context
0812 05:22:57.135929 (+ 65us) client_negotiation.cc:240] Negotiation successful
0812 05:22:57.135966 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":185,"mutex_wait_us":34,"thread_start_us":115,"threads_started":1}
I20260812 05:22:57.136325 24536 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1605964bd9443e7a6fa66525884030d" candidate_uuid: "323e33cb599a49dc8c4ad5ca105d6634" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d517dee1abe4f2e8070c046252ea7be" is_pre_election: true
I20260812 05:22:57.136382 24014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1605964bd9443e7a6fa66525884030d" candidate_uuid: "323e33cb599a49dc8c4ad5ca105d6634" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e183f30ebe8b419f8b2fdec556634e45" is_pre_election: true
I20260812 05:22:57.136552 24014 raft_consensus.cc:2454] T f1605964bd9443e7a6fa66525884030d P e183f30ebe8b419f8b2fdec556634e45 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 323e33cb599a49dc8c4ad5ca105d6634 in term 0.
I20260812 05:22:57.136575 24536 raft_consensus.cc:2454] T f1605964bd9443e7a6fa66525884030d P 4d517dee1abe4f2e8070c046252ea7be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 323e33cb599a49dc8c4ad5ca105d6634 in term 0.
I20260812 05:22:57.136772 24156 leader_election.cc:302] T f1605964bd9443e7a6fa66525884030d P 323e33cb599a49dc8c4ad5ca105d6634 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 323e33cb599a49dc8c4ad5ca105d6634, e183f30ebe8b419f8b2fdec556634e45; no voters:
I20260812 05:22:57.136909 24646 raft_consensus.cc:2781] T f1605964bd9443e7a6fa66525884030d P 323e33cb599a49dc8c4ad5ca105d6634 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 05:22:57.136966 24646 raft_consensus.cc:482] T f1605964bd9443e7a6fa66525884030d P 323e33cb599a49dc8c4ad5ca105d6634 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 05:22:57.136988 24646 raft_consensus.cc:3037] T f1605964bd9443e7a6fa66525884030d P 323e33cb599a49dc8c4ad5ca105d6634 [term 0 FOLLOWER]: Advancing to term 1
I20260812 05:22:57.137715 24646 raft_consensus.cc:504] T f1605964bd9443e7a6fa66525884030d P 323e33cb599a49dc8c4ad5ca105d6634 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "323e33cb599a49dc8c4ad5ca105d6634" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 40255 } } peers { permanent_uuid: "4d517dee1abe4f2e8070c046252ea7be" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 44797 } } peers { permanent_uuid: "e183f30ebe8b419f8b2fdec556634e45" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 44501 } }
I20260812 05:22:57.137828 24646 leader_election.cc:288] T f1605964bd9443e7a6fa66525884030d P 323e33cb599a49dc8c4ad5ca105d6634 [CANDIDATE]: Term 1 election: Requested vote from peers 4d517dee1abe4f2e8070c046252ea7be (127.31.100.3:44797), e183f30ebe8b419f8b2fdec556634e45 (127.31.100.1:44501)
I20260812 05:22:57.138083 24536 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1605964bd9443e7a6fa66525884030d" candidate_uuid: "323e33cb599a49dc8c4ad5ca105d6634" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d517dee1abe4f2e8070c046252ea7be"
I20260812 05:22:57.138083 24014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1605964bd9443e7a6fa66525884030d" candidate_uuid: "323e33cb599a49dc8c4ad5ca105d6634" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e183f30ebe8b419f8b2fdec556634e45"
I20260812 05:22:57.138180 24536 raft_consensus.cc:3037] T f1605964bd9443e7a6fa66525884030d P 4d517dee1abe4f2e8070c046252ea7be [term 0 FOLLOWER]: Advancing to term 1
I20260812 05:22:57.138180 24014 raft_consensus.cc:3037] T f1605964bd9443e7a6fa66525884030d P e183f30ebe8b419f8b2fdec556634e45 [term 0 FOLLOWER]: Advancing to term 1
I20260812 05:22:57.139014 24536 raft_consensus.cc:2454] T f1605964bd9443e7a6fa66525884030d P 4d517dee1abe4f2e8070c046252ea7be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 323e33cb599a49dc8c4ad5ca105d6634 in term 1.
I20260812 05:22:57.139024 24014 raft_consensus.cc:2454] T f1605964bd9443e7a6fa66525884030d P e183f30ebe8b419f8b2fdec556634e45 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 323e33cb599a49dc8c4ad5ca105d6634 in term 1.
I20260812 05:22:57.139206 24149 leader_election.cc:302] T f1605964bd9443e7a6fa66525884030d P 323e33cb599a49dc8c4ad5ca105d6634 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 323e33cb599a49dc8c4ad5ca105d6634, 4d517dee1abe4f2e8070c046252ea7be; no voters:
I20260812 05:22:57.139310 24646 raft_consensus.cc:2781] T f1605964bd9443e7a6fa66525884030d P 323e33cb599a49dc8c4ad5ca105d6634 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 05:22:57.139498 24646 raft_consensus.cc:686] T f1605964bd9443e7a6fa66525884030d P 323e33cb599a49dc8c4ad5ca105d6634 [term 1 LEADER]: Becoming Leader. State: Replica: 323e33cb599a49dc8c4ad5ca105d6634, State: Running, Role: LEADER
I20260812 05:22:57.139600 24646 consensus_queue.cc:237] T f1605964bd9443e7a6fa66525884030d P 323e33cb599a49dc8c4ad5ca105d6634 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "323e33cb599a49dc8c4ad5ca105d6634" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 40255 } } peers { permanent_uuid: "4d517dee1abe4f2e8070c046252ea7be" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 44797 } } peers { permanent_uuid: "e183f30ebe8b419f8b2fdec556634e45" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 44501 } }
I20260812 05:22:57.140429 23747 catalog_manager.cc:5495] T f1605964bd9443e7a6fa66525884030d P 323e33cb599a49dc8c4ad5ca105d6634 reported cstate change: term changed from 0 to 1, leader changed from <none> to 323e33cb599a49dc8c4ad5ca105d6634 (127.31.100.2). New cstate: current_term: 1 leader_uuid: "323e33cb599a49dc8c4ad5ca105d6634" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "323e33cb599a49dc8c4ad5ca105d6634" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 40255 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d517dee1abe4f2e8070c046252ea7be" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 44797 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e183f30ebe8b419f8b2fdec556634e45" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 44501 } health_report { overall_health: UNKNOWN } } }
I20260812 05:22:57.185683 24631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:22:57.183390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32898 (local address 127.31.100.2:40255)
0812 05:22:57.183453 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0812 05:22:57.183455 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 05:22:57.183463 (+ 8us) server_negotiation.cc:407] Connection header received
0812 05:22:57.183508 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:22:57.183509 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:22:57.183531 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:22:57.183591 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 05:22:57.183708 (+ 117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:22:57.184199 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:22:57.184902 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:22:57.185117 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:22:57.185174 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:22:57.185320 (+ 146us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 05:22:57.185425 (+ 105us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 05:22:57.185520 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0812 05:22:57.185559 (+ 39us) server_negotiation.cc:299] Negotiation successful
0812 05:22:57.185580 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260812 05:22:57.187487 24014 raft_consensus.cc:1261] T f1605964bd9443e7a6fa66525884030d P e183f30ebe8b419f8b2fdec556634e45 [term 1 FOLLOWER]: Refusing update from remote peer 323e33cb599a49dc8c4ad5ca105d6634: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 05:22:57.187503 24536 raft_consensus.cc:1261] T f1605964bd9443e7a6fa66525884030d P 4d517dee1abe4f2e8070c046252ea7be [term 1 FOLLOWER]: Refusing update from remote peer 323e33cb599a49dc8c4ad5ca105d6634: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 05:22:57.187793 24646 consensus_queue.cc:1035] T f1605964bd9443e7a6fa66525884030d P 323e33cb599a49dc8c4ad5ca105d6634 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e183f30ebe8b419f8b2fdec556634e45" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 44501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 05:22:57.187911 24649 consensus_queue.cc:1035] T f1605964bd9443e7a6fa66525884030d P 323e33cb599a49dc8c4ad5ca105d6634 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d517dee1abe4f2e8070c046252ea7be" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 44797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 05:22:57.190214 24651 mvcc.cc:205] Tried to move back new op lower bound from 7317553877757263872 to 7317553877564305408. Current Snapshot: MvccSnapshot[applied={T|T < 7317553877757263872}]
I20260812 05:22:57.190315 24654 mvcc.cc:205] Tried to move back new op lower bound from 7317553877757263872 to 7317553877564305408. Current Snapshot: MvccSnapshot[applied={T|T < 7317553877757263872}]
I20260812 05:22:57.192226 23747 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:59856:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 05:22:57.192323 23747 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:59856:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 05:22:57.193023 23747 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c866d37f66fb4b64bae37566f2baad8e: Sending DeleteTablet for 3 replicas of tablet f1605964bd9443e7a6fa66525884030d
I20260812 05:22:57.193322 24496 tablet_service.cc:1514] Processing DeleteTablet for tablet f1605964bd9443e7a6fa66525884030d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 05:22:57 UTC) from {username='kudu', principal='kudu/127.31.100.62@KRBTEST.COM'} at 127.0.0.1:34040
I20260812 05:22:57.193357 32144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0waFvT/build/release/bin/kudu with pid 23819
I20260812 05:22:57.193333 24232 tablet_service.cc:1514] Processing DeleteTablet for tablet f1605964bd9443e7a6fa66525884030d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 05:22:57 UTC) from {username='kudu', principal='kudu/127.31.100.62@KRBTEST.COM'} at 127.0.0.1:32892
I20260812 05:22:57.193360 23972 tablet_service.cc:1514] Processing DeleteTablet for tablet f1605964bd9443e7a6fa66525884030d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 05:22:57 UTC) from {username='kudu', principal='kudu/127.31.100.62@KRBTEST.COM'} at 127.0.0.1:54860
I20260812 05:22:57.205730 24671 tablet_replica.cc:330] T f1605964bd9443e7a6fa66525884030d P 4d517dee1abe4f2e8070c046252ea7be: stopping tablet replica
I20260812 05:22:57.205876 24671 raft_consensus.cc:2229] T f1605964bd9443e7a6fa66525884030d P 4d517dee1abe4f2e8070c046252ea7be [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 05:22:57.205996 24671 raft_consensus.cc:2258] T f1605964bd9443e7a6fa66525884030d P 4d517dee1abe4f2e8070c046252ea7be [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 05:22:57.206524 24671 ts_tablet_manager.cc:1905] T f1605964bd9443e7a6fa66525884030d P 4d517dee1abe4f2e8070c046252ea7be: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 05:22:57.207221 24668 tablet_replica.cc:330] T f1605964bd9443e7a6fa66525884030d P 323e33cb599a49dc8c4ad5ca105d6634: stopping tablet replica
I20260812 05:22:57.207366 24668 raft_consensus.cc:2229] T f1605964bd9443e7a6fa66525884030d P 323e33cb599a49dc8c4ad5ca105d6634 [term 1 LEADER]: Raft consensus shutting down.
I20260812 05:22:57.207513 24668 raft_consensus.cc:2258] T f1605964bd9443e7a6fa66525884030d P 323e33cb599a49dc8c4ad5ca105d6634 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 05:22:57.208016 24668 ts_tablet_manager.cc:1905] T f1605964bd9443e7a6fa66525884030d P 323e33cb599a49dc8c4ad5ca105d6634: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 05:22:57.208190 24671 ts_tablet_manager.cc:1918] T f1605964bd9443e7a6fa66525884030d P 4d517dee1abe4f2e8070c046252ea7be: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 05:22:57.208272 24671 log.cc:1196] T f1605964bd9443e7a6fa66525884030d P 4d517dee1abe4f2e8070c046252ea7be: Deleting WAL directory at /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786512017613366-32144-0/minicluster-data/ts-2/wal/wals/f1605964bd9443e7a6fa66525884030d
I20260812 05:22:57.208570 24671 ts_tablet_manager.cc:1939] T f1605964bd9443e7a6fa66525884030d P 4d517dee1abe4f2e8070c046252ea7be: Deleting consensus metadata
I20260812 05:22:57.209017 23722 catalog_manager.cc:4841] TS 4d517dee1abe4f2e8070c046252ea7be (127.31.100.3:44797): tablet f1605964bd9443e7a6fa66525884030d (table test-table [id=3e8453ed852a4078b5b017c8146f7b0f]) successfully deleted
I20260812 05:22:57.209751 24668 ts_tablet_manager.cc:1918] T f1605964bd9443e7a6fa66525884030d P 323e33cb599a49dc8c4ad5ca105d6634: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 05:22:57.209822 24668 log.cc:1196] T f1605964bd9443e7a6fa66525884030d P 323e33cb599a49dc8c4ad5ca105d6634: Deleting WAL directory at /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786512017613366-32144-0/minicluster-data/ts-1/wal/wals/f1605964bd9443e7a6fa66525884030d
I20260812 05:22:57.210065 24668 ts_tablet_manager.cc:1939] T f1605964bd9443e7a6fa66525884030d P 323e33cb599a49dc8c4ad5ca105d6634: Deleting consensus metadata
I20260812 05:22:57.210479 23723 catalog_manager.cc:4841] TS 323e33cb599a49dc8c4ad5ca105d6634 (127.31.100.2:40255): tablet f1605964bd9443e7a6fa66525884030d (table test-table [id=3e8453ed852a4078b5b017c8146f7b0f]) successfully deleted
W20260812 05:22:57.210937 23724 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.31.100.1:44501 (error 108)
I20260812 05:22:57.211274 32144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0waFvT/build/release/bin/kudu with pid 24104
I20260812 05:22:57.213023 24113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:22:57.211395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.1:44501 (local address 127.0.0.1:54870)
0812 05:22:57.212931 (+ 1536us) negotiation.cc:107] Waiting for socket to connect
0812 05:22:57.212986 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.100.1:44501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1506}
W20260812 05:22:57.217526 23724 catalog_manager.cc:4584] TS e183f30ebe8b419f8b2fdec556634e45 (127.31.100.1:44501): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f1605964bd9443e7a6fa66525884030d: Network error: Client connection negotiation failed: client connection to 127.31.100.1:44501: connect: Connection refused (error 111)
I20260812 05:22:57.223095 32144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0waFvT/build/release/bin/kudu with pid 24380
I20260812 05:22:57.231372 32144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0waFvT/build/release/bin/kudu with pid 23692
I20260812 05:22:57.240177 32144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0waFvT/build/release/bin/kudu
/tmp/dist-test-task0waFvT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786512017613366-32144-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786512017613366-32144-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786512017613366-32144-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786512017613366-32144-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.100.62:40577
--webserver_interface=127.31.100.62
--webserver_port=43853
--keytab_file=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.62.keytab
--principal=kudu/127.31.100.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.100.20:37413
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.100.62:40577
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786512017613366-32144-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786512017613366-32144-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786512017613366-32144-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 05:22:57.320045 24673 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 05:22:57.320232 24673 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 05:22:57.320252 24673 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 05:22:57.321306 24673 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 05:22:57.321338 24673 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 05:22:57.321352 24673 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 05:22:57.321363 24673 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 05:22:57.321378 24673 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 05:22:57.322789 24673 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.100.20:37413
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786512017613366-32144-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786512017613366-32144-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.100.62:40577
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.62.keytab
--principal=kudu/127.31.100.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.100.62:40577
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786512017613366-32144-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.100.62
--webserver_port=43853
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24673
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786512017613366-32144-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 04:20:34 UTC on ccycloud.quasar-phqdqp.root.comops.site
build id 4581871
I20260812 05:22:57.322976 24673 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 05:22:57.323148 24673 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 05:22:57.327706 24684 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 05:22:57.327756 24685 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 05:22:57.327806 24673 server_base.cc:1034] running on GCE node
W20260812 05:22:57.327706 24687 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 05:22:57.328241 24673 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 05:22:57.328477 24673 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 05:22:57.329605 24673 hybrid_clock.cc:648] HybridClock initialized: now 1786512177329594 us; error 29 us; skew 500 ppm
Aug 12 05:22:57 dist-test-slave-42z9 krb5kdc[22682](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512177, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.100.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 05:22:57.331830 24673 init.cc:377] Logged in from keytab as kudu/127.31.100.62@KRBTEST.COM (short username kudu)
I20260812 05:22:57.332250 24673 webserver.cc:460] Webserver started at http://127.31.100.62:43853/ using document root <none> and password file <none>
I20260812 05:22:57.332456 24673 fs_manager.cc:359] Metadata directory not provided
I20260812 05:22:57.332504 24673 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 05:22:57.333725 24673 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260812 05:22:57.334297 24700 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 05:22:57.334440 24673 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260812 05:22:57.334532 24673 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786512017613366-32144-0/minicluster-data/master-0/data,/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786512017613366-32144-0/minicluster-data/master-0/wal
uuid: "c866d37f66fb4b64bae37566f2baad8e"
format_stamp: "Formatted at 2026-08-12 05:22:55 on dist-test-slave-42z9"
server_key: "9d146bbc124be7926356b761a88170c3"
server_key_iv: "59d13cc9c7748afda4d9392b9aad4348"
server_key_version: "encryptionkey@0"
I20260812 05:22:57.334830 24673 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786512017613366-32144-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786512017613366-32144-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786512017613366-32144-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 05:22:57.344173 24673 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 05:22:57.345777 24673 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 05:22:57.345896 24673 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 05:22:57.349617 24673 rpc_server.cc:307] RPC server started. Bound to: 127.31.100.62:40577
I20260812 05:22:57.349647 24815 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.100.62:40577 every 8 connection(s)
I20260812 05:22:57.349979 24673 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786512017613366-32144-0/minicluster-data/master-0/data/info.pb
I20260812 05:22:57.351469 24817 sys_catalog.cc:263] Verifying existing consensus state
I20260812 05:22:57.352120 24817 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c866d37f66fb4b64bae37566f2baad8e: Bootstrap starting.
I20260812 05:22:57.354820 32144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0waFvT/build/release/bin/kudu as pid 24673
I20260812 05:22:57.355091 32144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0waFvT/build/release/bin/kudu
/tmp/dist-test-task0waFvT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786512017613366-32144-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786512017613366-32144-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786512017613366-32144-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786512017613366-32144-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.100.1:44501
--local_ip_for_outbound_sockets=127.31.100.1
--tserver_master_addrs=127.31.100.62:40577
--webserver_port=43341
--webserver_interface=127.31.100.1
--keytab_file=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.1.keytab
--principal=kudu/127.31.100.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.100.20:37413
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786512017613366-32144-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786512017613366-32144-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786512017613366-32144-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 05:22:57.360566 24817 log.cc:824] T 00000000000000000000000000000000 P c866d37f66fb4b64bae37566f2baad8e: Log is configured to *not* fsync() on all Append() calls
I20260812 05:22:57.362879 24817 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c866d37f66fb4b64bae37566f2baad8e: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 05:22:57.363157 24817 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c866d37f66fb4b64bae37566f2baad8e: Bootstrap complete.
I20260812 05:22:57.365201 24817 raft_consensus.cc:348] T 00000000000000000000000000000000 P c866d37f66fb4b64bae37566f2baad8e [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c866d37f66fb4b64bae37566f2baad8e" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 40577 } }
I20260812 05:22:57.365413 24817 raft_consensus.cc:729] T 00000000000000000000000000000000 P c866d37f66fb4b64bae37566f2baad8e [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: c866d37f66fb4b64bae37566f2baad8e, State: Initialized, Role: FOLLOWER
I20260812 05:22:57.365599 24817 consensus_queue.cc:260] T 00000000000000000000000000000000 P c866d37f66fb4b64bae37566f2baad8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c866d37f66fb4b64bae37566f2baad8e" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 40577 } }
I20260812 05:22:57.365679 24817 raft_consensus.cc:388] T 00000000000000000000000000000000 P c866d37f66fb4b64bae37566f2baad8e [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 05:22:57.365729 24817 raft_consensus.cc:482] T 00000000000000000000000000000000 P c866d37f66fb4b64bae37566f2baad8e [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 05:22:57.365775 24817 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c866d37f66fb4b64bae37566f2baad8e [term 2 FOLLOWER]: Advancing to term 3
I20260812 05:22:57.366699 24817 raft_consensus.cc:504] T 00000000000000000000000000000000 P c866d37f66fb4b64bae37566f2baad8e [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c866d37f66fb4b64bae37566f2baad8e" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 40577 } }
I20260812 05:22:57.366796 24817 leader_election.cc:302] T 00000000000000000000000000000000 P c866d37f66fb4b64bae37566f2baad8e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c866d37f66fb4b64bae37566f2baad8e; no voters:
I20260812 05:22:57.366952 24817 leader_election.cc:288] T 00000000000000000000000000000000 P c866d37f66fb4b64bae37566f2baad8e [CANDIDATE]: Term 3 election: Requested vote from peers
I20260812 05:22:57.367007 24822 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c866d37f66fb4b64bae37566f2baad8e [term 3 FOLLOWER]: Leader election won for term 3
I20260812 05:22:57.367159 24822 raft_consensus.cc:686] T 00000000000000000000000000000000 P c866d37f66fb4b64bae37566f2baad8e [term 3 LEADER]: Becoming Leader. State: Replica: c866d37f66fb4b64bae37566f2baad8e, State: Running, Role: LEADER
I20260812 05:22:57.367206 24817 sys_catalog.cc:564] T 00000000000000000000000000000000 P c866d37f66fb4b64bae37566f2baad8e [sys.catalog]: configured and running, proceeding with master startup.
I20260812 05:22:57.367264 24822 consensus_queue.cc:237] T 00000000000000000000000000000000 P c866d37f66fb4b64bae37566f2baad8e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c866d37f66fb4b64bae37566f2baad8e" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 40577 } }
I20260812 05:22:57.367492 24822 sys_catalog.cc:455] T 00000000000000000000000000000000 P c866d37f66fb4b64bae37566f2baad8e [sys.catalog]: SysCatalogTable state changed. Reason: New leader c866d37f66fb4b64bae37566f2baad8e. Latest consensus state: current_term: 3 leader_uuid: "c866d37f66fb4b64bae37566f2baad8e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c866d37f66fb4b64bae37566f2baad8e" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 40577 } } }
I20260812 05:22:57.367564 24822 sys_catalog.cc:458] T 00000000000000000000000000000000 P c866d37f66fb4b64bae37566f2baad8e [sys.catalog]: This master's current role is: LEADER
I20260812 05:22:57.367493 24825 sys_catalog.cc:455] T 00000000000000000000000000000000 P c866d37f66fb4b64bae37566f2baad8e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "c866d37f66fb4b64bae37566f2baad8e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c866d37f66fb4b64bae37566f2baad8e" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 40577 } } }
I20260812 05:22:57.367724 24825 sys_catalog.cc:458] T 00000000000000000000000000000000 P c866d37f66fb4b64bae37566f2baad8e [sys.catalog]: This master's current role is: LEADER
I20260812 05:22:57.367743 24832 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 05:22:57.368650 24832 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 05:22:57.368817 24832 catalog_manager.cc:1259] Loaded cluster ID: ece460ce7ad94ebfacddc2bb26fd457b
I20260812 05:22:57.369048 24832 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260812 05:22:57.369323 24832 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f81012289ad google::LogMessage::Fail() at ??:0
@ 0x7f810122b88a google::LogMessage::SendToLog() at ??:0
@ 0x7f81012284de google::LogMessage::Flush() at ??:0
@ 0x7f8101229a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f81091490f1 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7f81091528b9 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f81030f60d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f81030eb954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f8103f4f6db start_thread at ??:0
@ 0x7f80ffd2661f clone at ??:0
W20260812 05:22:57.450001 24820 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 05:22:57.450192 24820 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 05:22:57.450213 24820 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 05:22:57.451335 24820 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 05:22:57.451375 24820 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 05:22:57.451399 24820 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.100.1
I20260812 05:22:57.452782 24820 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.100.20:37413
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786512017613366-32144-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786512017613366-32144-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.1.keytab
--principal=kudu/127.31.100.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.100.1:44501
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786512017613366-32144-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.100.1
--webserver_port=43341
--tserver_master_addrs=127.31.100.62:40577
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24820
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.100.1
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786512017613366-32144-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 04:20:34 UTC on ccycloud.quasar-phqdqp.root.comops.site
build id 4581871
I20260812 05:22:57.452993 24820 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 05:22:57.453204 24820 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 05:22:57.459210 24885 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 05:22:57.459228 24889 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 05:22:57.459303 24884 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 05:22:57.459766 24820 server_base.cc:1034] running on GCE node
I20260812 05:22:57.460160 24820 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 05:22:57.460433 24820 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 05:22:57.461596 24820 hybrid_clock.cc:648] HybridClock initialized: now 1786512177461581 us; error 28 us; skew 500 ppm
Aug 12 05:22:57 dist-test-slave-42z9 krb5kdc[22682](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512177, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.100.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 05:22:57.464480 24820 init.cc:377] Logged in from keytab as kudu/127.31.100.1@KRBTEST.COM (short username kudu)
I20260812 05:22:57.464891 24820 webserver.cc:460] Webserver started at http://127.31.100.1:43341/ using document root <none> and password file <none>
I20260812 05:22:57.465094 24820 fs_manager.cc:359] Metadata directory not provided
I20260812 05:22:57.465148 24820 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 05:22:57.466491 24820 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260812 05:22:57.467149 24898 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 05:22:57.467298 24820 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260812 05:22:57.467382 24820 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786512017613366-32144-0/minicluster-data/ts-0/data,/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786512017613366-32144-0/minicluster-data/ts-0/wal
uuid: "e183f30ebe8b419f8b2fdec556634e45"
format_stamp: "Formatted at 2026-08-12 05:22:55 on dist-test-slave-42z9"
server_key: "79fc4ee8ece2e5b458a42b62eebdf8e6"
server_key_iv: "ce475b6e38a19f92ff73166e94d397a9"
server_key_version: "encryptionkey@0"
I20260812 05:22:57.467696 24820 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786512017613366-32144-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786512017613366-32144-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786512017613366-32144-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 05:22:57.477994 24820 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 05:22:57.479944 24820 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 05:22:57.480078 24820 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 05:22:57.480280 24820 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 05:22:57.480679 24908 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260812 05:22:57.482569 24820 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260812 05:22:57.482618 24820 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260812 05:22:57.482648 24820 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260812 05:22:57.483271 24820 ts_tablet_manager.cc:610] Registered 1 tablets
I20260812 05:22:57.483304 24820 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260812 05:22:57.483393 24908 tablet_bootstrap.cc:492] T f1605964bd9443e7a6fa66525884030d P e183f30ebe8b419f8b2fdec556634e45: Bootstrap starting.
I20260812 05:22:57.489878 24820 rpc_server.cc:307] RPC server started. Bound to: 127.31.100.1:44501
I20260812 05:22:57.489993 25087 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.100.1:44501 every 8 connection(s)
I20260812 05:22:57.490268 24820 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786512017613366-32144-0/minicluster-data/ts-0/data/info.pb
I20260812 05:22:57.490597 32144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0waFvT/build/release/bin/kudu as pid 24820
I20260812 05:22:57.490885 32144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0waFvT/build/release/bin/kudu
/tmp/dist-test-task0waFvT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786512017613366-32144-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786512017613366-32144-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786512017613366-32144-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786512017613366-32144-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.100.2:40255
--local_ip_for_outbound_sockets=127.31.100.2
--tserver_master_addrs=127.31.100.62:40577
--webserver_port=42543
--webserver_interface=127.31.100.2
--keytab_file=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.2.keytab
--principal=kudu/127.31.100.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.100.20:37413
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786512017613366-32144-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786512017613366-32144-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786512017613366-32144-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 05:22:57.496564 24908 log.cc:824] T f1605964bd9443e7a6fa66525884030d P e183f30ebe8b419f8b2fdec556634e45: Log is configured to *not* fsync() on all Append() calls
I20260812 05:22:57.499189 24908 tablet_bootstrap.cc:492] T f1605964bd9443e7a6fa66525884030d P e183f30ebe8b419f8b2fdec556634e45: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 05:22:57.499511 24908 tablet_bootstrap.cc:492] T f1605964bd9443e7a6fa66525884030d P e183f30ebe8b419f8b2fdec556634e45: Bootstrap complete.
I20260812 05:22:57.500620 24908 ts_tablet_manager.cc:1397] T f1605964bd9443e7a6fa66525884030d P e183f30ebe8b419f8b2fdec556634e45: Time spent bootstrapping tablet: real 0.017s user 0.012s sys 0.000s
I20260812 05:22:57.502077 24908 raft_consensus.cc:348] T f1605964bd9443e7a6fa66525884030d P e183f30ebe8b419f8b2fdec556634e45 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "323e33cb599a49dc8c4ad5ca105d6634" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 40255 } } peers { permanent_uuid: "4d517dee1abe4f2e8070c046252ea7be" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 44797 } } peers { permanent_uuid: "e183f30ebe8b419f8b2fdec556634e45" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 44501 } }
I20260812 05:22:57.502388 24908 raft_consensus.cc:729] T f1605964bd9443e7a6fa66525884030d P e183f30ebe8b419f8b2fdec556634e45 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e183f30ebe8b419f8b2fdec556634e45, State: Initialized, Role: FOLLOWER
I20260812 05:22:57.502549 24908 consensus_queue.cc:260] T f1605964bd9443e7a6fa66525884030d P e183f30ebe8b419f8b2fdec556634e45 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "323e33cb599a49dc8c4ad5ca105d6634" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 40255 } } peers { permanent_uuid: "4d517dee1abe4f2e8070c046252ea7be" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 44797 } } peers { permanent_uuid: "e183f30ebe8b419f8b2fdec556634e45" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 44501 } }
I20260812 05:22:57.502833 24908 ts_tablet_manager.cc:1428] T f1605964bd9443e7a6fa66525884030d P e183f30ebe8b419f8b2fdec556634e45: Time spent starting tablet: real 0.002s user 0.005s sys 0.000s
W20260812 05:22:57.552124 25096 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 05:22:57.490865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.62:40577 (local address 127.31.100.1:52141)
0812 05:22:57.491174 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0812 05:22:57.491207 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0812 05:22:57.491612 (+ 405us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:22:57.551938 (+ 60326us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.100.62:40577: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":74,"threads_started":1}
W20260812 05:22:57.552389 25090 heartbeater.cc:644] Failed to heartbeat to 127.31.100.62:40577 (0 consecutive failures): Network error: Failed to ping master at 127.31.100.62:40577: Client connection negotiation failed: client connection to 127.31.100.62:40577: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260812 05:22:57.552736 25096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:22:57.552630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.62:40577 (local address 127.31.100.1:50093)
0812 05:22:57.552678 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0812 05:22:57.552714 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.100.62:40577: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260812 05:22:57.553062 25096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:22:57.552984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.62:40577 (local address 127.31.100.1:50695)
0812 05:22:57.553023 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0812 05:22:57.553048 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.100.62:40577: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260812 05:22:57.553162 25090 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260812 05:22:57.575428 25098 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 05:22:57.575610 25098 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 05:22:57.575631 25098 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 05:22:57.576681 25098 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 05:22:57.576712 25098 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 05:22:57.576737 25098 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.100.2
I20260812 05:22:57.578181 25098 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.100.20:37413
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786512017613366-32144-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786512017613366-32144-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.2.keytab
--principal=kudu/127.31.100.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.100.2:40255
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786512017613366-32144-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.100.2
--webserver_port=42543
--tserver_master_addrs=127.31.100.62:40577
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25098
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.100.2
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786512017613366-32144-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 04:20:34 UTC on ccycloud.quasar-phqdqp.root.comops.site
build id 4581871
I20260812 05:22:57.578395 25098 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 05:22:57.578598 25098 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 05:22:57.583374 25120 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 05:22:57.583392 25112 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 05:22:57.583534 25114 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 05:22:57.583422 25098 server_base.cc:1034] running on GCE node
I20260812 05:22:57.583734 25098 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 05:22:57.583923 25098 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 05:22:57.585057 25098 hybrid_clock.cc:648] HybridClock initialized: now 1786512177585048 us; error 32 us; skew 500 ppm
Aug 12 05:22:57 dist-test-slave-42z9 krb5kdc[22682](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512177, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.100.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 05:22:57.587414 25098 init.cc:377] Logged in from keytab as kudu/127.31.100.2@KRBTEST.COM (short username kudu)
I20260812 05:22:57.587821 25098 webserver.cc:460] Webserver started at http://127.31.100.2:42543/ using document root <none> and password file <none>
I20260812 05:22:57.588011 25098 fs_manager.cc:359] Metadata directory not provided
I20260812 05:22:57.588061 25098 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 05:22:57.589250 25098 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260812 05:22:57.589816 25137 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 05:22:57.589946 25098 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260812 05:22:57.590027 25098 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786512017613366-32144-0/minicluster-data/ts-1/data,/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786512017613366-32144-0/minicluster-data/ts-1/wal
uuid: "323e33cb599a49dc8c4ad5ca105d6634"
format_stamp: "Formatted at 2026-08-12 05:22:55 on dist-test-slave-42z9"
server_key: "ec87d2ef04a1a1fe9b2bd7e62ab79afe"
server_key_iv: "673423269407aaced0bfb9334e1c7d54"
server_key_version: "encryptionkey@0"
I20260812 05:22:57.590328 25098 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786512017613366-32144-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786512017613366-32144-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786512017613366-32144-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 05:22:57.601533 25098 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 05:22:57.603163 25098 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 05:22:57.603291 25098 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 05:22:57.603487 25098 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 05:22:57.603760 25098 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 05:22:57.603794 25098 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 05:22:57.603820 25098 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 05:22:57.603837 25098 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 05:22:57.609517 25098 rpc_server.cc:307] RPC server started. Bound to: 127.31.100.2:40255
I20260812 05:22:57.609583 25353 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.100.2:40255 every 8 connection(s)
I20260812 05:22:57.609871 25098 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786512017613366-32144-0/minicluster-data/ts-1/data/info.pb
I20260812 05:22:57.610805 25358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:22:57.610455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.62:40577 (local address 127.31.100.2:59949)
0812 05:22:57.610610 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0812 05:22:57.610678 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.100.62:40577: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":75,"threads_started":1}
W20260812 05:22:57.611176 25354 heartbeater.cc:644] Failed to heartbeat to 127.31.100.62:40577 (0 consecutive failures): Network error: Failed to ping master at 127.31.100.62:40577: Client connection negotiation failed: client connection to 127.31.100.62:40577: connect: Connection refused (error 111)
I20260812 05:22:57.611403 25358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:22:57.611324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.62:40577 (local address 127.31.100.2:49755)
0812 05:22:57.611361 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0812 05:22:57.611386 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.100.62:40577: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
I20260812 05:22:57.611653 25358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:22:57.611590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.62:40577 (local address 127.31.100.2:35471)
0812 05:22:57.611621 (+ 31us) negotiation.cc:107] Waiting for socket to connect
0812 05:22:57.611640 (+ 19us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.100.62:40577: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
W20260812 05:22:57.611750 25354 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260812 05:22:57.617332 32144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0waFvT/build/release/bin/kudu as pid 25098
I20260812 05:22:57.617630 32144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0waFvT/build/release/bin/kudu
/tmp/dist-test-task0waFvT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786512017613366-32144-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786512017613366-32144-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786512017613366-32144-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786512017613366-32144-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.100.3:44797
--local_ip_for_outbound_sockets=127.31.100.3
--tserver_master_addrs=127.31.100.62:40577
--webserver_port=38583
--webserver_interface=127.31.100.3
--keytab_file=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.3.keytab
--principal=kudu/127.31.100.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.100.20:37413
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786512017613366-32144-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786512017613366-32144-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786512017613366-32144-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 05:22:57.699175 25359 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 05:22:57.699383 25359 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 05:22:57.699406 25359 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 05:22:57.700506 25359 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 05:22:57.700544 25359 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 05:22:57.700572 25359 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.100.3
I20260812 05:22:57.702092 25359 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.100.20:37413
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786512017613366-32144-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786512017613366-32144-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.3.keytab
--principal=kudu/127.31.100.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.100.3:44797
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786512017613366-32144-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.100.3
--webserver_port=38583
--tserver_master_addrs=127.31.100.62:40577
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25359
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.100.3
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786512017613366-32144-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 04:20:34 UTC on ccycloud.quasar-phqdqp.root.comops.site
build id 4581871
I20260812 05:22:57.702325 25359 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 05:22:57.702540 25359 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 05:22:57.707132 25369 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 05:22:57.707194 25366 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 05:22:57.707237 25367 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 05:22:57.707458 25359 server_base.cc:1034] running on GCE node
I20260812 05:22:57.707647 25359 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 05:22:57.707850 25359 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 05:22:57.708988 25359 hybrid_clock.cc:648] HybridClock initialized: now 1786512177708974 us; error 28 us; skew 500 ppm
Aug 12 05:22:57 dist-test-slave-42z9 krb5kdc[22682](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512177, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.100.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 05:22:57.711306 25359 init.cc:377] Logged in from keytab as kudu/127.31.100.3@KRBTEST.COM (short username kudu)
I20260812 05:22:57.711692 25359 webserver.cc:460] Webserver started at http://127.31.100.3:38583/ using document root <none> and password file <none>
I20260812 05:22:57.711860 25359 fs_manager.cc:359] Metadata directory not provided
I20260812 05:22:57.711898 25359 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 05:22:57.713089 25359 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260812 05:22:57.713608 25382 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 05:22:57.713761 25359 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260812 05:22:57.713845 25359 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786512017613366-32144-0/minicluster-data/ts-2/data,/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786512017613366-32144-0/minicluster-data/ts-2/wal
uuid: "4d517dee1abe4f2e8070c046252ea7be"
format_stamp: "Formatted at 2026-08-12 05:22:56 on dist-test-slave-42z9"
server_key: "3b6ae4bd9db96d6907c558c9f73f404b"
server_key_iv: "44679731b0da20a67ac4afea59c62f29"
server_key_version: "encryptionkey@0"
I20260812 05:22:57.714128 25359 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786512017613366-32144-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786512017613366-32144-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786512017613366-32144-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 05:22:57.726672 25359 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 05:22:57.728236 25359 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 05:22:57.728336 25359 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 05:22:57.728497 25359 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 05:22:57.728737 25359 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 05:22:57.728766 25359 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 05:22:57.728785 25359 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 05:22:57.728801 25359 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 05:22:57.734088 25359 rpc_server.cc:307] RPC server started. Bound to: 127.31.100.3:44797
I20260812 05:22:57.734151 25581 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.100.3:44797 every 8 connection(s)
I20260812 05:22:57.734426 25359 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786512017613366-32144-0/minicluster-data/ts-2/data/info.pb
I20260812 05:22:57.735306 25585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:22:57.734974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.62:40577 (local address 127.31.100.3:53343)
0812 05:22:57.735110 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0812 05:22:57.735178 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.100.62:40577: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":69,"threads_started":1}
W20260812 05:22:57.735661 25583 heartbeater.cc:644] Failed to heartbeat to 127.31.100.62:40577 (0 consecutive failures): Network error: Failed to ping master at 127.31.100.62:40577: Client connection negotiation failed: client connection to 127.31.100.62:40577: connect: Connection refused (error 111)
I20260812 05:22:57.735898 25585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:22:57.735812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.62:40577 (local address 127.31.100.3:54833)
0812 05:22:57.735850 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0812 05:22:57.735874 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.100.62:40577: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
I20260812 05:22:57.736150 25585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:22:57.736079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.62:40577 (local address 127.31.100.3:53193)
0812 05:22:57.736110 (+ 31us) negotiation.cc:107] Waiting for socket to connect
0812 05:22:57.736131 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.100.62:40577: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":20}
W20260812 05:22:57.736240 25583 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260812 05:22:57.743057 32144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0waFvT/build/release/bin/kudu as pid 25359
I20260812 05:22:58.553718 25593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:22:58.553518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.62:40577 (local address 127.31.100.1:44453)
0812 05:22:58.553627 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0812 05:22:58.553660 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.100.62:40577: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":60,"threads_started":1}
I20260812 05:22:58.612210 25594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:22:58.612005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.62:40577 (local address 127.31.100.2:58683)
0812 05:22:58.612114 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0812 05:22:58.612156 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.100.62:40577: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":57,"threads_started":1}
I20260812 05:22:58.736697 25595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:22:58.736494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.62:40577 (local address 127.31.100.3:46483)
0812 05:22:58.736617 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0812 05:22:58.736647 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.100.62:40577: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":74,"threads_started":1}
I20260812 05:22:59.107358 25596 raft_consensus.cc:482] T f1605964bd9443e7a6fa66525884030d P e183f30ebe8b419f8b2fdec556634e45 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 05:22:59.107492 25596 raft_consensus.cc:504] T f1605964bd9443e7a6fa66525884030d P e183f30ebe8b419f8b2fdec556634e45 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "323e33cb599a49dc8c4ad5ca105d6634" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 40255 } } peers { permanent_uuid: "4d517dee1abe4f2e8070c046252ea7be" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 44797 } } peers { permanent_uuid: "e183f30ebe8b419f8b2fdec556634e45" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 44501 } }
I20260812 05:22:59.107776 25596 leader_election.cc:288] T f1605964bd9443e7a6fa66525884030d P e183f30ebe8b419f8b2fdec556634e45 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 323e33cb599a49dc8c4ad5ca105d6634 (127.31.100.2:40255), 4d517dee1abe4f2e8070c046252ea7be (127.31.100.3:44797)
Aug 12 05:22:59 dist-test-slave-42z9 krb5kdc[22682](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512177, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.100.1@KRBTEST.COM for kudu/127.31.100.3@KRBTEST.COM
Aug 12 05:22:59 dist-test-slave-42z9 krb5kdc[22682](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512177, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.100.1@KRBTEST.COM for kudu/127.31.100.2@KRBTEST.COM
I20260812 05:22:59.117594 25602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:22:59.108037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.100.1:56563 (local address 127.31.100.3:44797)
0812 05:22:59.108206 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0812 05:22:59.108218 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0812 05:22:59.108251 (+ 33us) server_negotiation.cc:407] Connection header received
0812 05:22:59.108710 (+ 459us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:22:59.108713 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:22:59.108760 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:22:59.108887 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:22:59.109648 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:22:59.110498 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:22:59.111496 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:22:59.111873 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:22:59.111973 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:22:59.114385 (+ 2412us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 05:22:59.114388 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 05:22:59.114389 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 05:22:59.114398 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0812 05:22:59.116113 (+ 1715us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:22:59.116459 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:22:59.116462 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:22:59.116462 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:22:59.116543 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:22:59.116712 (+ 169us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:22:59.116713 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:22:59.116713 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:22:59.116932 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 05:22:59.117000 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0812 05:22:59.117363 (+ 363us) server_negotiation.cc:299] Negotiation successful
0812 05:22:59.117517 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":48,"threads_started":1}
I20260812 05:22:59.117621 25597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:22:59.107925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.2:40255 (local address 127.31.100.1:51317)
0812 05:22:59.108070 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0812 05:22:59.108081 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0812 05:22:59.108142 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:22:59.108938 (+ 796us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:22:59.108940 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:22:59.109316 (+ 376us) client_negotiation.cc:189] Negotiated authn=SASL
0812 05:22:59.109545 (+ 229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:22:59.109551 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:22:59.110588 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:22:59.110589 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:22:59.111396 (+ 807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:22:59.111399 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:22:59.111957 (+ 558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:22:59.111958 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:22:59.112065 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:22:59.112624 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 05:22:59.112628 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0812 05:22:59.114296 (+ 1668us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 05:22:59.116031 (+ 1735us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 05:22:59.116057 (+ 26us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 05:22:59.116059 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 05:22:59.116323 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 05:22:59.116548 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 05:22:59.116549 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 05:22:59.116549 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 05:22:59.116651 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 05:22:59.116997 (+ 346us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 05:22:59.117003 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 05:22:59.117144 (+ 141us) client_negotiation.cc:769] Sending connection context
0812 05:22:59.117240 (+ 96us) client_negotiation.cc:240] Negotiation successful
0812 05:22:59.117527 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"mutex_wait_us":67,"thread_start_us":72,"threads_started":1}
I20260812 05:22:59.117631 25603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:22:59.108116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.100.1:51317 (local address 127.31.100.2:40255)
0812 05:22:59.108259 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0812 05:22:59.108268 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 05:22:59.108295 (+ 27us) server_negotiation.cc:407] Connection header received
0812 05:22:59.108710 (+ 415us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:22:59.108712 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:22:59.108752 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:22:59.108873 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:22:59.109660 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:22:59.110493 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:22:59.111486 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:22:59.111873 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:22:59.111981 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:22:59.114379 (+ 2398us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 05:22:59.114383 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 05:22:59.114383 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 05:22:59.114393 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0812 05:22:59.115921 (+ 1528us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:22:59.116403 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:22:59.116406 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:22:59.116407 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:22:59.116491 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:22:59.116707 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:22:59.116708 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:22:59.116708 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:22:59.116933 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 05:22:59.117002 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0812 05:22:59.117360 (+ 358us) server_negotiation.cc:299] Negotiation successful
0812 05:22:59.117518 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":43,"threads_started":1}
I20260812 05:22:59.117851 25601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:22:59.107930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.3:44797 (local address 127.31.100.1:56563)
0812 05:22:59.108105 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0812 05:22:59.108114 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0812 05:22:59.108154 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:22:59.108938 (+ 784us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:22:59.108941 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:22:59.109317 (+ 376us) client_negotiation.cc:189] Negotiated authn=SASL
0812 05:22:59.109545 (+ 228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:22:59.109551 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:22:59.110606 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:22:59.110606 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:22:59.111395 (+ 789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:22:59.111398 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:22:59.111959 (+ 561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:22:59.111960 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:22:59.112067 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:22:59.112587 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 05:22:59.112604 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0812 05:22:59.114056 (+ 1452us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 05:22:59.116175 (+ 2119us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 05:22:59.116176 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 05:22:59.116177 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 05:22:59.116393 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 05:22:59.116601 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 05:22:59.116602 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 05:22:59.116603 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 05:22:59.116663 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 05:22:59.116995 (+ 332us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 05:22:59.117003 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 05:22:59.117156 (+ 153us) client_negotiation.cc:769] Sending connection context
0812 05:22:59.117241 (+ 85us) client_negotiation.cc:240] Negotiation successful
0812 05:22:59.117772 (+ 531us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163,"mutex_wait_us":466,"thread_start_us":79,"threads_started":1}
W20260812 05:22:59.118569 24901 leader_election.cc:341] T f1605964bd9443e7a6fa66525884030d P e183f30ebe8b419f8b2fdec556634e45 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 4d517dee1abe4f2e8070c046252ea7be (127.31.100.3:44797): Not found: Tablet not found: f1605964bd9443e7a6fa66525884030d
W20260812 05:22:59.118634 24902 leader_election.cc:341] T f1605964bd9443e7a6fa66525884030d P e183f30ebe8b419f8b2fdec556634e45 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 323e33cb599a49dc8c4ad5ca105d6634 (127.31.100.2:40255): Not found: Tablet not found: f1605964bd9443e7a6fa66525884030d
I20260812 05:22:59.118710 24902 leader_election.cc:302] T f1605964bd9443e7a6fa66525884030d P e183f30ebe8b419f8b2fdec556634e45 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e183f30ebe8b419f8b2fdec556634e45; no voters: 323e33cb599a49dc8c4ad5ca105d6634, 4d517dee1abe4f2e8070c046252ea7be
I20260812 05:22:59.118824 25596 raft_consensus.cc:2726] T f1605964bd9443e7a6fa66525884030d P e183f30ebe8b419f8b2fdec556634e45 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260812 05:22:59.554244 25601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:22:59.554123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.62:40577 (local address 127.31.100.1:54283)
0812 05:22:59.554179 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0812 05:22:59.554205 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.100.62:40577: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260812 05:22:59.612795 25605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:22:59.612599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.62:40577 (local address 127.31.100.2:32943)
0812 05:22:59.612715 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0812 05:22:59.612749 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.100.62:40577: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":60,"threads_started":1}
I20260812 05:22:59.737267 25606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:22:59.737083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.62:40577 (local address 127.31.100.3:38057)
0812 05:22:59.737199 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0812 05:22:59.737229 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.100.62:40577: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":63,"threads_started":1}
I20260812 05:23:00.554894 25608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:23:00.554665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.62:40577 (local address 127.31.100.1:51391)
0812 05:23:00.554807 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0812 05:23:00.554848 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.100.62:40577: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":67,"threads_started":1}
I20260812 05:23:00.613384 25611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:23:00.613188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.62:40577 (local address 127.31.100.2:57797)
0812 05:23:00.613291 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0812 05:23:00.613332 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.100.62:40577: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83,"thread_start_us":48,"threads_started":1}
I20260812 05:23:00.737821 25612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:23:00.737615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.62:40577 (local address 127.31.100.3:60837)
0812 05:23:00.737749 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0812 05:23:00.737780 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.100.62:40577: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":81,"threads_started":1}
I20260812 05:23:01.316083 25614 raft_consensus.cc:482] T f1605964bd9443e7a6fa66525884030d P e183f30ebe8b419f8b2fdec556634e45 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 05:23:01.316179 25614 raft_consensus.cc:504] T f1605964bd9443e7a6fa66525884030d P e183f30ebe8b419f8b2fdec556634e45 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "323e33cb599a49dc8c4ad5ca105d6634" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 40255 } } peers { permanent_uuid: "4d517dee1abe4f2e8070c046252ea7be" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 44797 } } peers { permanent_uuid: "e183f30ebe8b419f8b2fdec556634e45" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 44501 } }
I20260812 05:23:01.316320 25614 leader_election.cc:288] T f1605964bd9443e7a6fa66525884030d P e183f30ebe8b419f8b2fdec556634e45 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 323e33cb599a49dc8c4ad5ca105d6634 (127.31.100.2:40255), 4d517dee1abe4f2e8070c046252ea7be (127.31.100.3:44797)
W20260812 05:23:01.316788 24901 leader_election.cc:341] T f1605964bd9443e7a6fa66525884030d P e183f30ebe8b419f8b2fdec556634e45 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 4d517dee1abe4f2e8070c046252ea7be (127.31.100.3:44797): Not found: Tablet not found: f1605964bd9443e7a6fa66525884030d
W20260812 05:23:01.316895 24902 leader_election.cc:341] T f1605964bd9443e7a6fa66525884030d P e183f30ebe8b419f8b2fdec556634e45 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 323e33cb599a49dc8c4ad5ca105d6634 (127.31.100.2:40255): Not found: Tablet not found: f1605964bd9443e7a6fa66525884030d
I20260812 05:23:01.316956 24902 leader_election.cc:302] T f1605964bd9443e7a6fa66525884030d P e183f30ebe8b419f8b2fdec556634e45 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e183f30ebe8b419f8b2fdec556634e45; no voters: 323e33cb599a49dc8c4ad5ca105d6634, 4d517dee1abe4f2e8070c046252ea7be
I20260812 05:23:01.317018 25614 raft_consensus.cc:2726] T f1605964bd9443e7a6fa66525884030d P e183f30ebe8b419f8b2fdec556634e45 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260812 05:23:01.555588 25616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:23:01.555349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.62:40577 (local address 127.31.100.1:58173)
0812 05:23:01.555507 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0812 05:23:01.555540 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.100.62:40577: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":73,"threads_started":1}
I20260812 05:23:01.614022 25619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:23:01.613806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.62:40577 (local address 127.31.100.2:60769)
0812 05:23:01.613934 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0812 05:23:01.613968 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.100.62:40577: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":69,"threads_started":1}
I20260812 05:23:01.738444 25620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:23:01.738202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.62:40577 (local address 127.31.100.3:33347)
0812 05:23:01.738328 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0812 05:23:01.738384 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.100.62:40577: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":62,"threads_started":1}
I20260812 05:23:02.556265 25621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:23:02.556063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.62:40577 (local address 127.31.100.1:48115)
0812 05:23:02.556183 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0812 05:23:02.556221 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.100.62:40577: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":65,"threads_started":1}
I20260812 05:23:02.614656 25623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:23:02.614432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.62:40577 (local address 127.31.100.2:59391)
0812 05:23:02.614573 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0812 05:23:02.614614 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.100.62:40577: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":61,"threads_started":1}
I20260812 05:23:02.739070 25625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:23:02.738856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.62:40577 (local address 127.31.100.3:52605)
0812 05:23:02.738988 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0812 05:23:02.739026 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.100.62:40577: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":63,"threads_started":1}
I20260812 05:23:03.396531 25627 raft_consensus.cc:482] T f1605964bd9443e7a6fa66525884030d P e183f30ebe8b419f8b2fdec556634e45 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 05:23:03.396624 25627 raft_consensus.cc:504] T f1605964bd9443e7a6fa66525884030d P e183f30ebe8b419f8b2fdec556634e45 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "323e33cb599a49dc8c4ad5ca105d6634" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 40255 } } peers { permanent_uuid: "4d517dee1abe4f2e8070c046252ea7be" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 44797 } } peers { permanent_uuid: "e183f30ebe8b419f8b2fdec556634e45" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 44501 } }
I20260812 05:23:03.396760 25627 leader_election.cc:288] T f1605964bd9443e7a6fa66525884030d P e183f30ebe8b419f8b2fdec556634e45 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 323e33cb599a49dc8c4ad5ca105d6634 (127.31.100.2:40255), 4d517dee1abe4f2e8070c046252ea7be (127.31.100.3:44797)
W20260812 05:23:03.397136 24902 leader_election.cc:341] T f1605964bd9443e7a6fa66525884030d P e183f30ebe8b419f8b2fdec556634e45 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 323e33cb599a49dc8c4ad5ca105d6634 (127.31.100.2:40255): Not found: Tablet not found: f1605964bd9443e7a6fa66525884030d
W20260812 05:23:03.397224 24901 leader_election.cc:341] T f1605964bd9443e7a6fa66525884030d P e183f30ebe8b419f8b2fdec556634e45 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 4d517dee1abe4f2e8070c046252ea7be (127.31.100.3:44797): Not found: Tablet not found: f1605964bd9443e7a6fa66525884030d
I20260812 05:23:03.397264 24901 leader_election.cc:302] T f1605964bd9443e7a6fa66525884030d P e183f30ebe8b419f8b2fdec556634e45 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e183f30ebe8b419f8b2fdec556634e45; no voters: 323e33cb599a49dc8c4ad5ca105d6634, 4d517dee1abe4f2e8070c046252ea7be
I20260812 05:23:03.397328 25627 raft_consensus.cc:2726] T f1605964bd9443e7a6fa66525884030d P e183f30ebe8b419f8b2fdec556634e45 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260812 05:23:03.556934 25628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:23:03.556737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.62:40577 (local address 127.31.100.1:34987)
0812 05:23:03.556854 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0812 05:23:03.556889 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.100.62:40577: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":64,"threads_started":1}
I20260812 05:23:03.615267 25630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:23:03.615087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.62:40577 (local address 127.31.100.2:51277)
0812 05:23:03.615198 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0812 05:23:03.615226 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.100.62:40577: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":60,"threads_started":1}
I20260812 05:23:03.739682 25631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:23:03.739478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.62:40577 (local address 127.31.100.3:41723)
0812 05:23:03.739610 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0812 05:23:03.739639 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.100.62:40577: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":80,"threads_started":1}
I20260812 05:23:04.557610 25633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:23:04.557343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.62:40577 (local address 127.31.100.1:35141)
0812 05:23:04.557520 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0812 05:23:04.557560 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.100.62:40577: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":114,"threads_started":1}
I20260812 05:23:04.615859 25635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:23:04.615682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.62:40577 (local address 127.31.100.2:58761)
0812 05:23:04.615787 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0812 05:23:04.615819 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.100.62:40577: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91,"thread_start_us":56,"threads_started":1}
I20260812 05:23:04.740314 25636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:23:04.740116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.62:40577 (local address 127.31.100.3:40497)
0812 05:23:04.740238 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0812 05:23:04.740272 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.100.62:40577: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":62,"threads_started":1}
I20260812 05:23:05.558270 25637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:23:05.558045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.62:40577 (local address 127.31.100.1:59447)
0812 05:23:05.558185 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0812 05:23:05.558222 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.100.62:40577: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":63,"threads_started":1}
I20260812 05:23:05.616389 25638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:23:05.616183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.62:40577 (local address 127.31.100.2:42485)
0812 05:23:05.616309 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0812 05:23:05.616343 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.100.62:40577: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":64,"threads_started":1}
I20260812 05:23:05.740903 25639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:23:05.740705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.62:40577 (local address 127.31.100.3:55929)
0812 05:23:05.740829 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0812 05:23:05.740862 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.100.62:40577: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":77,"threads_started":1}
I20260812 05:23:06.502265 25640 raft_consensus.cc:482] T f1605964bd9443e7a6fa66525884030d P e183f30ebe8b419f8b2fdec556634e45 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 05:23:06.502362 25640 raft_consensus.cc:504] T f1605964bd9443e7a6fa66525884030d P e183f30ebe8b419f8b2fdec556634e45 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "323e33cb599a49dc8c4ad5ca105d6634" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 40255 } } peers { permanent_uuid: "4d517dee1abe4f2e8070c046252ea7be" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 44797 } } peers { permanent_uuid: "e183f30ebe8b419f8b2fdec556634e45" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 44501 } }
I20260812 05:23:06.502650 25640 leader_election.cc:288] T f1605964bd9443e7a6fa66525884030d P e183f30ebe8b419f8b2fdec556634e45 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 323e33cb599a49dc8c4ad5ca105d6634 (127.31.100.2:40255), 4d517dee1abe4f2e8070c046252ea7be (127.31.100.3:44797)
W20260812 05:23:06.502976 24902 leader_election.cc:341] T f1605964bd9443e7a6fa66525884030d P e183f30ebe8b419f8b2fdec556634e45 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 323e33cb599a49dc8c4ad5ca105d6634 (127.31.100.2:40255): Not found: Tablet not found: f1605964bd9443e7a6fa66525884030d
W20260812 05:23:06.503067 24901 leader_election.cc:341] T f1605964bd9443e7a6fa66525884030d P e183f30ebe8b419f8b2fdec556634e45 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 4d517dee1abe4f2e8070c046252ea7be (127.31.100.3:44797): Not found: Tablet not found: f1605964bd9443e7a6fa66525884030d
I20260812 05:23:06.503126 24901 leader_election.cc:302] T f1605964bd9443e7a6fa66525884030d P e183f30ebe8b419f8b2fdec556634e45 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e183f30ebe8b419f8b2fdec556634e45; no voters: 323e33cb599a49dc8c4ad5ca105d6634, 4d517dee1abe4f2e8070c046252ea7be
I20260812 05:23:06.503190 25640 raft_consensus.cc:2726] T f1605964bd9443e7a6fa66525884030d P e183f30ebe8b419f8b2fdec556634e45 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260812 05:23:06.558867 25642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:23:06.558669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.62:40577 (local address 127.31.100.1:58353)
0812 05:23:06.558789 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0812 05:23:06.558822 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.100.62:40577: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":64,"threads_started":1}
I20260812 05:23:06.616976 25644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:23:06.616747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.62:40577 (local address 127.31.100.2:49003)
0812 05:23:06.616891 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0812 05:23:06.616935 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.100.62:40577: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":68,"threads_started":1}
I20260812 05:23:06.741568 25645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:23:06.741340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.62:40577 (local address 127.31.100.3:43833)
0812 05:23:06.741491 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0812 05:23:06.741525 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.100.62:40577: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":77,"threads_started":1}
I20260812 05:23:07.559557 25646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:23:07.559316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.62:40577 (local address 127.31.100.1:53575)
0812 05:23:07.559451 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0812 05:23:07.559508 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.100.62:40577: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":73,"threads_started":1}
I20260812 05:23:07.617589 25648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:23:07.617311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.62:40577 (local address 127.31.100.2:35697)
0812 05:23:07.617507 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0812 05:23:07.617547 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.100.62:40577: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":84,"threads_started":1}
I20260812 05:23:07.742128 25649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:23:07.741934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.62:40577 (local address 127.31.100.3:36737)
0812 05:23:07.742052 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0812 05:23:07.742086 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.100.62:40577: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":63,"threads_started":1}
I20260812 05:23:08.560175 25650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:23:08.559966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.62:40577 (local address 127.31.100.1:34141)
0812 05:23:08.560091 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0812 05:23:08.560127 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.100.62:40577: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":65,"threads_started":1}
I20260812 05:23:08.618137 25651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:23:08.617944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.62:40577 (local address 127.31.100.2:54425)
0812 05:23:08.618063 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0812 05:23:08.618097 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.100.62:40577: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":55,"threads_started":1}
I20260812 05:23:08.742683 25653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:23:08.742486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.62:40577 (local address 127.31.100.3:50947)
0812 05:23:08.742607 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0812 05:23:08.742640 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.100.62:40577: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":64,"threads_started":1}
I20260812 05:23:09.560801 25654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:23:09.560586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.62:40577 (local address 127.31.100.1:40815)
0812 05:23:09.560721 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0812 05:23:09.560754 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.100.62:40577: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":79,"threads_started":1}
I20260812 05:23:09.618700 25657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:23:09.618492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.62:40577 (local address 127.31.100.2:44739)
0812 05:23:09.618625 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0812 05:23:09.618658 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.100.62:40577: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":78,"threads_started":1}
I20260812 05:23:09.743281 25658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:23:09.743074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.62:40577 (local address 127.31.100.3:60139)
0812 05:23:09.743202 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0812 05:23:09.743236 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.100.62:40577: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":66,"threads_started":1}
I20260812 05:23:09.961187 25659 raft_consensus.cc:482] T f1605964bd9443e7a6fa66525884030d P e183f30ebe8b419f8b2fdec556634e45 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 05:23:09.961287 25659 raft_consensus.cc:504] T f1605964bd9443e7a6fa66525884030d P e183f30ebe8b419f8b2fdec556634e45 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "323e33cb599a49dc8c4ad5ca105d6634" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 40255 } } peers { permanent_uuid: "4d517dee1abe4f2e8070c046252ea7be" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 44797 } } peers { permanent_uuid: "e183f30ebe8b419f8b2fdec556634e45" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 44501 } }
I20260812 05:23:09.961485 25659 leader_election.cc:288] T f1605964bd9443e7a6fa66525884030d P e183f30ebe8b419f8b2fdec556634e45 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 323e33cb599a49dc8c4ad5ca105d6634 (127.31.100.2:40255), 4d517dee1abe4f2e8070c046252ea7be (127.31.100.3:44797)
W20260812 05:23:09.961930 24901 leader_election.cc:341] T f1605964bd9443e7a6fa66525884030d P e183f30ebe8b419f8b2fdec556634e45 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 4d517dee1abe4f2e8070c046252ea7be (127.31.100.3:44797): Not found: Tablet not found: f1605964bd9443e7a6fa66525884030d
W20260812 05:23:09.962020 24902 leader_election.cc:341] T f1605964bd9443e7a6fa66525884030d P e183f30ebe8b419f8b2fdec556634e45 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 323e33cb599a49dc8c4ad5ca105d6634 (127.31.100.2:40255): Not found: Tablet not found: f1605964bd9443e7a6fa66525884030d
I20260812 05:23:09.962064 24902 leader_election.cc:302] T f1605964bd9443e7a6fa66525884030d P e183f30ebe8b419f8b2fdec556634e45 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e183f30ebe8b419f8b2fdec556634e45; no voters: 323e33cb599a49dc8c4ad5ca105d6634, 4d517dee1abe4f2e8070c046252ea7be
I20260812 05:23:09.962133 25659 raft_consensus.cc:2726] T f1605964bd9443e7a6fa66525884030d P e183f30ebe8b419f8b2fdec556634e45 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260812 05:23:10.561434 25660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:23:10.561225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.62:40577 (local address 127.31.100.1:44881)
0812 05:23:10.561352 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0812 05:23:10.561388 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.100.62:40577: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":65,"threads_started":1}
I20260812 05:23:10.619271 25662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:23:10.619092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.62:40577 (local address 127.31.100.2:37049)
0812 05:23:10.619198 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0812 05:23:10.619231 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.100.62:40577: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":59,"threads_started":1}
I20260812 05:23:10.743870 25663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:23:10.743680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.62:40577 (local address 127.31.100.3:50217)
0812 05:23:10.743794 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0812 05:23:10.743826 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.100.62:40577: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":61,"threads_started":1}
I20260812 05:23:11.562166 25664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:23:11.561936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.62:40577 (local address 127.31.100.1:47121)
0812 05:23:11.562064 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0812 05:23:11.562104 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.100.62:40577: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":73,"threads_started":1}
I20260812 05:23:11.619881 25665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:23:11.619680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.62:40577 (local address 127.31.100.2:34143)
0812 05:23:11.619803 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0812 05:23:11.619838 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.100.62:40577: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":64,"threads_started":1}
I20260812 05:23:11.744433 25666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:23:11.744242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.62:40577 (local address 127.31.100.3:44425)
0812 05:23:11.744359 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0812 05:23:11.744388 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.100.62:40577: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":62,"threads_started":1}
I20260812 05:23:12.562852 25668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:23:12.562619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.62:40577 (local address 127.31.100.1:52675)
0812 05:23:12.562770 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0812 05:23:12.562806 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.100.62:40577: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":67,"threads_started":1}
I20260812 05:23:12.620565 25669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:23:12.620337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.62:40577 (local address 127.31.100.2:39955)
0812 05:23:12.620487 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0812 05:23:12.620522 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.100.62:40577: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":75,"threads_started":1}
I20260812 05:23:12.745061 25670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:23:12.744868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.62:40577 (local address 127.31.100.3:45483)
0812 05:23:12.744986 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0812 05:23:12.745017 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.100.62:40577: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":64,"threads_started":1}
I20260812 05:23:12.752511 32144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0waFvT/build/release/bin/kudu with pid 24820
I20260812 05:23:12.769248 32144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0waFvT/build/release/bin/kudu with pid 25098
I20260812 05:23:12.777172 32144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0waFvT/build/release/bin/kudu with pid 25359
2026-08-12T05:23:12Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (18575 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786512017613366-32144-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 05:23:12 dist-test-slave-42z9 krb5kdc[25677](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 05:23:12 dist-test-slave-42z9 krb5kdc[25677](info): set up 2 sockets
Aug 12 05:23:12 dist-test-slave-42z9 krb5kdc[25677](info): commencing operation
krb5kdc: starting...
W20260812 05:23:13.941319 32144 mini_kdc.cc:121] Time spent starting KDC: real 1.138s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786512017613366-32144-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786512017613366-32144-0/krb5kdc/test-user.keytab.
Aug 12 05:23:14 dist-test-slave-42z9 krb5kdc[25677](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512194, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T05:23:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T05:23:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.100.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.100.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.100.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.62.keytab.
Entry for principal kudu/127.31.100.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.62.keytab.
I20260812 05:23:14.056044 32144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0waFvT/build/release/bin/kudu
/tmp/dist-test-task0waFvT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786512017613366-32144-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786512017613366-32144-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786512017613366-32144-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786512017613366-32144-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.100.62:37129
--webserver_interface=127.31.100.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.62.keytab
--principal=kudu/127.31.100.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.100.20:45999
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.100.62:37129
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786512017613366-32144-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786512017613366-32144-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786512017613366-32144-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 05:23:14.136771 25699 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 05:23:14.136981 25699 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 05:23:14.137012 25699 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 05:23:14.138195 25699 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260812 05:23:14.138236 25699 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260812 05:23:14.138252 25699 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260812 05:23:14.138267 25699 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 05:23:14.138284 25699 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 05:23:14.138311 25699 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 05:23:14.138336 25699 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 05:23:14.138358 25699 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 05:23:14.139894 25699 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.100.20:45999
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786512017613366-32144-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786512017613366-32144-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.31.100.62:37129
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.62.keytab
--principal=kudu/127.31.100.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.100.62:37129
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786512017613366-32144-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.100.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25699
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786512017613366-32144-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 04:20:34 UTC on ccycloud.quasar-phqdqp.root.comops.site
build id 4581871
I20260812 05:23:14.140149 25699 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 05:23:14.140347 25699 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 05:23:14.145231 25719 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 05:23:14.145323 25699 server_base.cc:1034] running on GCE node
W20260812 05:23:14.145223 25722 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 05:23:14.145224 25715 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 05:23:14.145709 25699 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 05:23:14.146000 25699 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 05:23:14.147130 25699 hybrid_clock.cc:648] HybridClock initialized: now 1786512194147117 us; error 32 us; skew 500 ppm
Aug 12 05:23:14 dist-test-slave-42z9 krb5kdc[25677](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512194, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.100.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 05:23:14.149473 25699 init.cc:377] Logged in from keytab as kudu/127.31.100.62@KRBTEST.COM (short username kudu)
I20260812 05:23:14.149879 25699 webserver.cc:460] Webserver started at http://127.31.100.62:43699/ using document root <none> and password file <none>
I20260812 05:23:14.150134 25699 fs_manager.cc:359] Metadata directory not provided
I20260812 05:23:14.150185 25699 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 05:23:14.150309 25699 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 05:23:14.151340 25699 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786512017613366-32144-0/minicluster-data/master-0/data/instance:
uuid: "a2ede0bc8af24780a89103af190035b8"
format_stamp: "Formatted at 2026-08-12 05:23:14 on dist-test-slave-42z9"
server_key: "a0b2f7fff55987d6788b7a426193d037"
server_key_iv: "bf518a542e138ca511cd0bbdc2024334"
server_key_version: "encryptionkey@0"
I20260812 05:23:14.151672 25699 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786512017613366-32144-0/minicluster-data/master-0/wal/instance:
uuid: "a2ede0bc8af24780a89103af190035b8"
format_stamp: "Formatted at 2026-08-12 05:23:14 on dist-test-slave-42z9"
server_key: "a0b2f7fff55987d6788b7a426193d037"
server_key_iv: "bf518a542e138ca511cd0bbdc2024334"
server_key_version: "encryptionkey@0"
I20260812 05:23:14.152880 25699 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260812 05:23:14.153512 25733 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 05:23:14.153658 25699 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260812 05:23:14.153744 25699 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786512017613366-32144-0/minicluster-data/master-0/data,/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786512017613366-32144-0/minicluster-data/master-0/wal
uuid: "a2ede0bc8af24780a89103af190035b8"
format_stamp: "Formatted at 2026-08-12 05:23:14 on dist-test-slave-42z9"
server_key: "a0b2f7fff55987d6788b7a426193d037"
server_key_iv: "bf518a542e138ca511cd0bbdc2024334"
server_key_version: "encryptionkey@0"
I20260812 05:23:14.153810 25699 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786512017613366-32144-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786512017613366-32144-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786512017613366-32144-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 05:23:14.170858 25699 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 05:23:14.172868 25699 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 05:23:14.173010 25699 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 05:23:14.176757 25699 rpc_server.cc:307] RPC server started. Bound to: 127.31.100.62:37129
I20260812 05:23:14.176806 25827 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.100.62:37129 every 8 connection(s)
I20260812 05:23:14.177121 25699 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786512017613366-32144-0/minicluster-data/master-0/data/info.pb
I20260812 05:23:14.177968 25829 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 05:23:14.180259 25829 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a2ede0bc8af24780a89103af190035b8: Bootstrap starting.
I20260812 05:23:14.180833 25829 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a2ede0bc8af24780a89103af190035b8: Neither blocks nor log segments found. Creating new log.
I20260812 05:23:14.180886 32144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0waFvT/build/release/bin/kudu as pid 25699
I20260812 05:23:14.180975 32144 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786512017613366-32144-0/minicluster-data/master-0/wal/instance
I20260812 05:23:14.181137 25829 log.cc:824] T 00000000000000000000000000000000 P a2ede0bc8af24780a89103af190035b8: Log is configured to *not* fsync() on all Append() calls
I20260812 05:23:14.181164 32144 external_mini_cluster.cc:1428] Setting key 8a98ddd5df73adfc52a150684bb9fa1d
I20260812 05:23:14.181958 25829 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a2ede0bc8af24780a89103af190035b8: No bootstrap required, opened a new log
I20260812 05:23:14.183866 25829 raft_consensus.cc:348] T 00000000000000000000000000000000 P a2ede0bc8af24780a89103af190035b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2ede0bc8af24780a89103af190035b8" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 37129 } }
I20260812 05:23:14.183996 25829 raft_consensus.cc:374] T 00000000000000000000000000000000 P a2ede0bc8af24780a89103af190035b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 05:23:14.184027 25829 raft_consensus.cc:729] T 00000000000000000000000000000000 P a2ede0bc8af24780a89103af190035b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2ede0bc8af24780a89103af190035b8, State: Initialized, Role: FOLLOWER
I20260812 05:23:14.184151 25829 consensus_queue.cc:260] T 00000000000000000000000000000000 P a2ede0bc8af24780a89103af190035b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2ede0bc8af24780a89103af190035b8" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 37129 } }
I20260812 05:23:14.184221 25829 raft_consensus.cc:388] T 00000000000000000000000000000000 P a2ede0bc8af24780a89103af190035b8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 05:23:14.184242 25829 raft_consensus.cc:482] T 00000000000000000000000000000000 P a2ede0bc8af24780a89103af190035b8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 05:23:14.184283 25829 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a2ede0bc8af24780a89103af190035b8 [term 0 FOLLOWER]: Advancing to term 1
I20260812 05:23:14.184813 25829 raft_consensus.cc:504] T 00000000000000000000000000000000 P a2ede0bc8af24780a89103af190035b8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2ede0bc8af24780a89103af190035b8" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 37129 } }
I20260812 05:23:14.184909 25829 leader_election.cc:302] T 00000000000000000000000000000000 P a2ede0bc8af24780a89103af190035b8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a2ede0bc8af24780a89103af190035b8; no voters:
I20260812 05:23:14.185060 25829 leader_election.cc:288] T 00000000000000000000000000000000 P a2ede0bc8af24780a89103af190035b8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 05:23:14.185107 25835 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a2ede0bc8af24780a89103af190035b8 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 05:23:14.185316 25829 sys_catalog.cc:564] T 00000000000000000000000000000000 P a2ede0bc8af24780a89103af190035b8 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 05:23:14.185316 25835 raft_consensus.cc:686] T 00000000000000000000000000000000 P a2ede0bc8af24780a89103af190035b8 [term 1 LEADER]: Becoming Leader. State: Replica: a2ede0bc8af24780a89103af190035b8, State: Running, Role: LEADER
I20260812 05:23:14.185554 25835 consensus_queue.cc:237] T 00000000000000000000000000000000 P a2ede0bc8af24780a89103af190035b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2ede0bc8af24780a89103af190035b8" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 37129 } }
I20260812 05:23:14.185949 25837 sys_catalog.cc:455] T 00000000000000000000000000000000 P a2ede0bc8af24780a89103af190035b8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a2ede0bc8af24780a89103af190035b8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2ede0bc8af24780a89103af190035b8" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 37129 } } }
I20260812 05:23:14.186029 25837 sys_catalog.cc:458] T 00000000000000000000000000000000 P a2ede0bc8af24780a89103af190035b8 [sys.catalog]: This master's current role is: LEADER
I20260812 05:23:14.186247 25839 sys_catalog.cc:455] T 00000000000000000000000000000000 P a2ede0bc8af24780a89103af190035b8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a2ede0bc8af24780a89103af190035b8. Latest consensus state: current_term: 1 leader_uuid: "a2ede0bc8af24780a89103af190035b8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2ede0bc8af24780a89103af190035b8" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 37129 } } }
I20260812 05:23:14.186358 25839 sys_catalog.cc:458] T 00000000000000000000000000000000 P a2ede0bc8af24780a89103af190035b8 [sys.catalog]: This master's current role is: LEADER
I20260812 05:23:14.186750 25858 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 05:23:14.187288 25858 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Aug 12 05:23:14 dist-test-slave-42z9 krb5kdc[25677](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512194, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.100.62@KRBTEST.COM
I20260812 05:23:14.188874 25858 catalog_manager.cc:1347] Generated new cluster ID: 946abde4098a48769bf5aba02bbd3087
I20260812 05:23:14.188926 25858 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 05:23:14.191632 25834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:23:14.181605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36000 (local address 127.31.100.62:37129)
0812 05:23:14.181752 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0812 05:23:14.181761 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 05:23:14.181799 (+ 38us) server_negotiation.cc:407] Connection header received
0812 05:23:14.182258 (+ 459us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:23:14.182265 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:23:14.182314 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:23:14.182436 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:23:14.183070 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:23:14.183965 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:23:14.184791 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:23:14.185343 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:23:14.185814 (+ 471us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:23:14.187991 (+ 2177us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 05:23:14.187996 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 05:23:14.187996 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 05:23:14.188008 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0812 05:23:14.190057 (+ 2049us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:23:14.190436 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:23:14.190439 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:23:14.190439 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:23:14.190550 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:23:14.190725 (+ 175us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:23:14.190726 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:23:14.190726 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:23:14.191019 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 05:23:14.191162 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0812 05:23:14.191269 (+ 107us) server_negotiation.cc:299] Negotiation successful
0812 05:23:14.191402 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99,"thread_start_us":60,"threads_started":1}
I20260812 05:23:14.226541 25858 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 05:23:14.227075 25858 catalog_manager.cc:1504] Loading token signing keys...
I20260812 05:23:14.233059 25858 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a2ede0bc8af24780a89103af190035b8: Generated new TSK 0
I20260812 05:23:14.233224 25858 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.31.100.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.100.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.100.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.1.keytab.
Entry for principal kudu/127.31.100.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.1.keytab.
I20260812 05:23:14.267884 32144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0waFvT/build/release/bin/kudu
/tmp/dist-test-task0waFvT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786512017613366-32144-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786512017613366-32144-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786512017613366-32144-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786512017613366-32144-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.100.1:0
--local_ip_for_outbound_sockets=127.31.100.1
--webserver_interface=127.31.100.1
--webserver_port=0
--tserver_master_addrs=127.31.100.62:37129
--keytab_file=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.1.keytab
--principal=kudu/127.31.100.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.100.20:45999
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786512017613366-32144-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786512017613366-32144-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786512017613366-32144-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 05:23:14.356936 25876 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 05:23:14.357132 25876 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 05:23:14.357153 25876 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 05:23:14.358256 25876 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 05:23:14.358292 25876 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 05:23:14.358317 25876 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.100.1
I20260812 05:23:14.359758 25876 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.100.20:45999
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786512017613366-32144-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786512017613366-32144-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.1.keytab
--principal=kudu/127.31.100.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.100.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786512017613366-32144-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.100.1
--webserver_port=0
--tserver_master_addrs=127.31.100.62:37129
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25876
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.100.1
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786512017613366-32144-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 04:20:34 UTC on ccycloud.quasar-phqdqp.root.comops.site
build id 4581871
I20260812 05:23:14.359959 25876 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 05:23:14.360132 25876 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 05:23:14.364948 25887 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 05:23:14.365024 25876 server_base.cc:1034] running on GCE node
W20260812 05:23:14.364984 25890 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 05:23:14.364938 25888 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 05:23:14.365428 25876 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 05:23:14.365690 25876 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 05:23:14.366828 25876 hybrid_clock.cc:648] HybridClock initialized: now 1786512194366808 us; error 35 us; skew 500 ppm
Aug 12 05:23:14 dist-test-slave-42z9 krb5kdc[25677](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512194, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.100.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 05:23:14.369184 25876 init.cc:377] Logged in from keytab as kudu/127.31.100.1@KRBTEST.COM (short username kudu)
I20260812 05:23:14.369683 25876 webserver.cc:460] Webserver started at http://127.31.100.1:43255/ using document root <none> and password file <none>
I20260812 05:23:14.369899 25876 fs_manager.cc:359] Metadata directory not provided
I20260812 05:23:14.369947 25876 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 05:23:14.370074 25876 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 05:23:14.371069 25876 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786512017613366-32144-0/minicluster-data/ts-0/data/instance:
uuid: "3f75ed97c1aa4752b59f6a1551419390"
format_stamp: "Formatted at 2026-08-12 05:23:14 on dist-test-slave-42z9"
server_key: "1ff21222bd34bb43ebd41add3e2d3a43"
server_key_iv: "a68b8fa533bdffcbc4072c9da796609f"
server_key_version: "encryptionkey@0"
I20260812 05:23:14.371397 25876 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786512017613366-32144-0/minicluster-data/ts-0/wal/instance:
uuid: "3f75ed97c1aa4752b59f6a1551419390"
format_stamp: "Formatted at 2026-08-12 05:23:14 on dist-test-slave-42z9"
server_key: "1ff21222bd34bb43ebd41add3e2d3a43"
server_key_iv: "a68b8fa533bdffcbc4072c9da796609f"
server_key_version: "encryptionkey@0"
I20260812 05:23:14.372575 25876 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260812 05:23:14.373256 25905 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 05:23:14.373401 25876 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260812 05:23:14.373520 25876 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786512017613366-32144-0/minicluster-data/ts-0/data,/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786512017613366-32144-0/minicluster-data/ts-0/wal
uuid: "3f75ed97c1aa4752b59f6a1551419390"
format_stamp: "Formatted at 2026-08-12 05:23:14 on dist-test-slave-42z9"
server_key: "1ff21222bd34bb43ebd41add3e2d3a43"
server_key_iv: "a68b8fa533bdffcbc4072c9da796609f"
server_key_version: "encryptionkey@0"
I20260812 05:23:14.373587 25876 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786512017613366-32144-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786512017613366-32144-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786512017613366-32144-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 05:23:14.385597 25876 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 05:23:14.387465 25876 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 05:23:14.387594 25876 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 05:23:14.387789 25876 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 05:23:14.388072 25876 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 05:23:14.388105 25876 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 05:23:14.388132 25876 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 05:23:14.388149 25876 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 05:23:14.393635 25876 rpc_server.cc:307] RPC server started. Bound to: 127.31.100.1:37667
I20260812 05:23:14.393702 26111 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.100.1:37667 every 8 connection(s)
I20260812 05:23:14.393973 25876 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786512017613366-32144-0/minicluster-data/ts-0/data/info.pb
Aug 12 05:23:14 dist-test-slave-42z9 krb5kdc[25677](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512194, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.100.1@KRBTEST.COM for kudu/127.31.100.62@KRBTEST.COM
I20260812 05:23:14.402299 25834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:23:14.394592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.100.1:60697 (local address 127.31.100.62:37129)
0812 05:23:14.394640 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0812 05:23:14.394642 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 05:23:14.394809 (+ 167us) server_negotiation.cc:407] Connection header received
0812 05:23:14.395255 (+ 446us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:23:14.395256 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:23:14.395267 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:23:14.395316 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:23:14.396191 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:23:14.396717 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:23:14.397677 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:23:14.397893 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:23:14.397955 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:23:14.399932 (+ 1977us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 05:23:14.399934 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 05:23:14.399934 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 05:23:14.399940 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0812 05:23:14.401153 (+ 1213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:23:14.401546 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:23:14.401547 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:23:14.401547 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:23:14.401596 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:23:14.401848 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:23:14.401849 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:23:14.401849 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:23:14.401941 (+ 92us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 05:23:14.401983 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0812 05:23:14.402174 (+ 191us) server_negotiation.cc:299] Negotiation successful
0812 05:23:14.402236 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260812 05:23:14.402508 26117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:23:14.394570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.62:37129 (local address 127.31.100.1:60697)
0812 05:23:14.394711 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0812 05:23:14.394745 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0812 05:23:14.395183 (+ 438us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:23:14.395367 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:23:14.395370 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:23:14.395918 (+ 548us) client_negotiation.cc:189] Negotiated authn=SASL
0812 05:23:14.396101 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:23:14.396106 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:23:14.396792 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:23:14.396792 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:23:14.397598 (+ 806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:23:14.397600 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:23:14.397958 (+ 358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:23:14.397959 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:23:14.398057 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:23:14.398513 (+ 456us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 05:23:14.398535 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0812 05:23:14.399849 (+ 1314us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 05:23:14.401222 (+ 1373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 05:23:14.401228 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 05:23:14.401229 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 05:23:14.401489 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 05:23:14.401674 (+ 185us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 05:23:14.401675 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 05:23:14.401675 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 05:23:14.401796 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 05:23:14.401985 (+ 189us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 05:23:14.401988 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 05:23:14.402089 (+ 101us) client_negotiation.cc:769] Sending connection context
0812 05:23:14.402163 (+ 74us) client_negotiation.cc:240] Negotiation successful
0812 05:23:14.402282 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":78,"threads_started":1}
I20260812 05:23:14.402973 26112 heartbeater.cc:347] Connected to a master server at 127.31.100.62:37129
I20260812 05:23:14.403059 26112 heartbeater.cc:464] Registering TS with master...
I20260812 05:23:14.403219 26112 heartbeater.cc:511] Master 127.31.100.62:37129 requested a full tablet report, sending...
I20260812 05:23:14.403224 32144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0waFvT/build/release/bin/kudu as pid 25876
I20260812 05:23:14.403312 32144 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786512017613366-32144-0/minicluster-data/ts-0/wal/instance
I20260812 05:23:14.403538 32144 external_mini_cluster.cc:1428] Setting key 35d83808971e9169c1fe30f714071069
I20260812 05:23:14.403715 25762 ts_manager.cc:194] Registered new tserver with Master: 3f75ed97c1aa4752b59f6a1551419390 (127.31.100.1:37667)
I20260812 05:23:14.404472 25762 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.100.1@KRBTEST.COM'} at 127.31.100.1:60697
WARNING: no policy specified for kudu/127.31.100.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.100.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.100.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.2.keytab.
Entry for principal kudu/127.31.100.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.2.keytab.
I20260812 05:23:14.426115 32144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0waFvT/build/release/bin/kudu
/tmp/dist-test-task0waFvT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786512017613366-32144-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786512017613366-32144-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786512017613366-32144-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786512017613366-32144-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.100.2:0
--local_ip_for_outbound_sockets=127.31.100.2
--webserver_interface=127.31.100.2
--webserver_port=0
--tserver_master_addrs=127.31.100.62:37129
--keytab_file=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.2.keytab
--principal=kudu/127.31.100.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.100.20:45999
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786512017613366-32144-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786512017613366-32144-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786512017613366-32144-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 05:23:14.505715 26121 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 05:23:14.505913 26121 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 05:23:14.505935 26121 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 05:23:14.507043 26121 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 05:23:14.507082 26121 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 05:23:14.507107 26121 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.100.2
I20260812 05:23:14.508527 26121 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.100.20:45999
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786512017613366-32144-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786512017613366-32144-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.2.keytab
--principal=kudu/127.31.100.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.100.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786512017613366-32144-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.100.2
--webserver_port=0
--tserver_master_addrs=127.31.100.62:37129
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26121
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.100.2
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786512017613366-32144-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 04:20:34 UTC on ccycloud.quasar-phqdqp.root.comops.site
build id 4581871
I20260812 05:23:14.508737 26121 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 05:23:14.508949 26121 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 05:23:14.513707 26137 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 05:23:14.513715 26132 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 05:23:14.513803 26121 server_base.cc:1034] running on GCE node
W20260812 05:23:14.513751 26131 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 05:23:14.514061 26121 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 05:23:14.514288 26121 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 05:23:14.515430 26121 hybrid_clock.cc:648] HybridClock initialized: now 1786512194515410 us; error 31 us; skew 500 ppm
Aug 12 05:23:14 dist-test-slave-42z9 krb5kdc[25677](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512194, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.100.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 05:23:14.517954 26121 init.cc:377] Logged in from keytab as kudu/127.31.100.2@KRBTEST.COM (short username kudu)
I20260812 05:23:14.518324 26121 webserver.cc:460] Webserver started at http://127.31.100.2:40853/ using document root <none> and password file <none>
I20260812 05:23:14.518496 26121 fs_manager.cc:359] Metadata directory not provided
I20260812 05:23:14.518533 26121 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 05:23:14.518623 26121 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 05:23:14.519584 26121 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786512017613366-32144-0/minicluster-data/ts-1/data/instance:
uuid: "edf25888a7e648348650523730de9401"
format_stamp: "Formatted at 2026-08-12 05:23:14 on dist-test-slave-42z9"
server_key: "e08482d6fcbcc44b05a705170d41aa7f"
server_key_iv: "f08036fffbeee9c4ac9bcf06ad123959"
server_key_version: "encryptionkey@0"
I20260812 05:23:14.519876 26121 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786512017613366-32144-0/minicluster-data/ts-1/wal/instance:
uuid: "edf25888a7e648348650523730de9401"
format_stamp: "Formatted at 2026-08-12 05:23:14 on dist-test-slave-42z9"
server_key: "e08482d6fcbcc44b05a705170d41aa7f"
server_key_iv: "f08036fffbeee9c4ac9bcf06ad123959"
server_key_version: "encryptionkey@0"
I20260812 05:23:14.521059 26121 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260812 05:23:14.521720 26146 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 05:23:14.521859 26121 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260812 05:23:14.521942 26121 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786512017613366-32144-0/minicluster-data/ts-1/data,/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786512017613366-32144-0/minicluster-data/ts-1/wal
uuid: "edf25888a7e648348650523730de9401"
format_stamp: "Formatted at 2026-08-12 05:23:14 on dist-test-slave-42z9"
server_key: "e08482d6fcbcc44b05a705170d41aa7f"
server_key_iv: "f08036fffbeee9c4ac9bcf06ad123959"
server_key_version: "encryptionkey@0"
I20260812 05:23:14.521992 26121 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786512017613366-32144-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786512017613366-32144-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786512017613366-32144-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 05:23:14.539209 26121 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 05:23:14.541227 26121 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 05:23:14.541333 26121 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 05:23:14.541549 26121 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 05:23:14.541855 26121 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 05:23:14.541891 26121 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 05:23:14.541911 26121 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 05:23:14.541927 26121 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 05:23:14.547139 26121 rpc_server.cc:307] RPC server started. Bound to: 127.31.100.2:39191
I20260812 05:23:14.547209 26353 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.100.2:39191 every 8 connection(s)
I20260812 05:23:14.547461 26121 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786512017613366-32144-0/minicluster-data/ts-1/data/info.pb
I20260812 05:23:14.550489 32144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0waFvT/build/release/bin/kudu as pid 26121
I20260812 05:23:14.550570 32144 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786512017613366-32144-0/minicluster-data/ts-1/wal/instance
I20260812 05:23:14.550747 32144 external_mini_cluster.cc:1428] Setting key caaea8fcd696ee612f8d2f3d276b8055
Aug 12 05:23:14 dist-test-slave-42z9 krb5kdc[25677](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512194, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.100.2@KRBTEST.COM for kudu/127.31.100.62@KRBTEST.COM
I20260812 05:23:14.556610 25834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:23:14.548092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.100.2:48325 (local address 127.31.100.62:37129)
0812 05:23:14.548133 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0812 05:23:14.548135 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 05:23:14.548330 (+ 195us) server_negotiation.cc:407] Connection header received
0812 05:23:14.548787 (+ 457us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:23:14.548788 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:23:14.548797 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:23:14.548851 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:23:14.549759 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:23:14.550254 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:23:14.551417 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:23:14.551696 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:23:14.551776 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:23:14.553900 (+ 2124us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 05:23:14.553902 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 05:23:14.553902 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 05:23:14.553910 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0812 05:23:14.555155 (+ 1245us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:23:14.555672 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:23:14.555673 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:23:14.555674 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:23:14.555732 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:23:14.555992 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:23:14.555992 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:23:14.555993 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:23:14.556094 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 05:23:14.556140 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0812 05:23:14.556453 (+ 313us) server_negotiation.cc:299] Negotiation successful
0812 05:23:14.556545 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260812 05:23:14.556835 26356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:23:14.548058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.62:37129 (local address 127.31.100.2:48325)
0812 05:23:14.548214 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0812 05:23:14.548253 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0812 05:23:14.548712 (+ 459us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:23:14.548924 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:23:14.548927 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:23:14.549488 (+ 561us) client_negotiation.cc:189] Negotiated authn=SASL
0812 05:23:14.549692 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:23:14.549697 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:23:14.550331 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:23:14.550332 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:23:14.551314 (+ 982us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:23:14.551317 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:23:14.551779 (+ 462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:23:14.551780 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:23:14.551895 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:23:14.552330 (+ 435us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 05:23:14.552337 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0812 05:23:14.553814 (+ 1477us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 05:23:14.555237 (+ 1423us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 05:23:14.555247 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 05:23:14.555249 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 05:23:14.555566 (+ 317us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 05:23:14.555793 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 05:23:14.555794 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 05:23:14.555794 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 05:23:14.555933 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 05:23:14.556145 (+ 212us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 05:23:14.556150 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 05:23:14.556309 (+ 159us) client_negotiation.cc:769] Sending connection context
0812 05:23:14.556439 (+ 130us) client_negotiation.cc:240] Negotiation successful
0812 05:23:14.556604 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":69,"threads_started":1}
I20260812 05:23:14.557329 26354 heartbeater.cc:347] Connected to a master server at 127.31.100.62:37129
I20260812 05:23:14.557421 26354 heartbeater.cc:464] Registering TS with master...
I20260812 05:23:14.557626 26354 heartbeater.cc:511] Master 127.31.100.62:37129 requested a full tablet report, sending...
I20260812 05:23:14.558015 25762 ts_manager.cc:194] Registered new tserver with Master: edf25888a7e648348650523730de9401 (127.31.100.2:39191)
I20260812 05:23:14.558522 25762 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.100.2@KRBTEST.COM'} at 127.31.100.2:48325
WARNING: no policy specified for kudu/127.31.100.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.100.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.100.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.3.keytab.
Entry for principal kudu/127.31.100.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.3.keytab.
I20260812 05:23:14.575732 32144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0waFvT/build/release/bin/kudu
/tmp/dist-test-task0waFvT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786512017613366-32144-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786512017613366-32144-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786512017613366-32144-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786512017613366-32144-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.100.3:0
--local_ip_for_outbound_sockets=127.31.100.3
--webserver_interface=127.31.100.3
--webserver_port=0
--tserver_master_addrs=127.31.100.62:37129
--keytab_file=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.3.keytab
--principal=kudu/127.31.100.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.100.20:45999
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786512017613366-32144-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786512017613366-32144-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786512017613366-32144-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 05:23:14.655452 26360 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 05:23:14.655642 26360 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 05:23:14.655663 26360 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 05:23:14.656766 26360 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 05:23:14.656808 26360 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 05:23:14.656832 26360 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.100.3
I20260812 05:23:14.658263 26360 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.100.20:45999
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786512017613366-32144-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786512017613366-32144-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.3.keytab
--principal=kudu/127.31.100.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.100.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786512017613366-32144-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.100.3
--webserver_port=0
--tserver_master_addrs=127.31.100.62:37129
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26360
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.100.3
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786512017613366-32144-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 04:20:34 UTC on ccycloud.quasar-phqdqp.root.comops.site
build id 4581871
I20260812 05:23:14.658452 26360 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 05:23:14.658628 26360 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 05:23:14.663311 26372 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 05:23:14.663326 26369 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 05:23:14.663309 26370 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 05:23:14.663516 26360 server_base.cc:1034] running on GCE node
I20260812 05:23:14.663673 26360 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 05:23:14.663883 26360 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 05:23:14.665023 26360 hybrid_clock.cc:648] HybridClock initialized: now 1786512194665006 us; error 27 us; skew 500 ppm
Aug 12 05:23:14 dist-test-slave-42z9 krb5kdc[25677](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512194, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.100.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 05:23:14.667372 26360 init.cc:377] Logged in from keytab as kudu/127.31.100.3@KRBTEST.COM (short username kudu)
I20260812 05:23:14.667773 26360 webserver.cc:460] Webserver started at http://127.31.100.3:38899/ using document root <none> and password file <none>
I20260812 05:23:14.667948 26360 fs_manager.cc:359] Metadata directory not provided
I20260812 05:23:14.667987 26360 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 05:23:14.668085 26360 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 05:23:14.669068 26360 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786512017613366-32144-0/minicluster-data/ts-2/data/instance:
uuid: "df8db312fb6c4c57aa51f490c25fa09f"
format_stamp: "Formatted at 2026-08-12 05:23:14 on dist-test-slave-42z9"
server_key: "9ce08b8ccb671a16de628348a0a75d2a"
server_key_iv: "186a8874839af3036fc47945c19039fb"
server_key_version: "encryptionkey@0"
I20260812 05:23:14.669353 26360 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786512017613366-32144-0/minicluster-data/ts-2/wal/instance:
uuid: "df8db312fb6c4c57aa51f490c25fa09f"
format_stamp: "Formatted at 2026-08-12 05:23:14 on dist-test-slave-42z9"
server_key: "9ce08b8ccb671a16de628348a0a75d2a"
server_key_iv: "186a8874839af3036fc47945c19039fb"
server_key_version: "encryptionkey@0"
I20260812 05:23:14.670688 26360 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260812 05:23:14.671298 26388 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 05:23:14.671475 26360 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260812 05:23:14.671558 26360 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786512017613366-32144-0/minicluster-data/ts-2/data,/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786512017613366-32144-0/minicluster-data/ts-2/wal
uuid: "df8db312fb6c4c57aa51f490c25fa09f"
format_stamp: "Formatted at 2026-08-12 05:23:14 on dist-test-slave-42z9"
server_key: "9ce08b8ccb671a16de628348a0a75d2a"
server_key_iv: "186a8874839af3036fc47945c19039fb"
server_key_version: "encryptionkey@0"
I20260812 05:23:14.671624 26360 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786512017613366-32144-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786512017613366-32144-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786512017613366-32144-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 05:23:14.688896 26360 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 05:23:14.690723 26360 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 05:23:14.690867 26360 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 05:23:14.691085 26360 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 05:23:14.691387 26360 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 05:23:14.691421 26360 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 05:23:14.691442 26360 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 05:23:14.691453 26360 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 05:23:14.697208 26360 rpc_server.cc:307] RPC server started. Bound to: 127.31.100.3:45871
I20260812 05:23:14.697264 26578 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.100.3:45871 every 8 connection(s)
I20260812 05:23:14.697641 26360 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786512017613366-32144-0/minicluster-data/ts-2/data/info.pb
I20260812 05:23:14.700469 32144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0waFvT/build/release/bin/kudu as pid 26360
I20260812 05:23:14.700556 32144 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786512017613366-32144-0/minicluster-data/ts-2/wal/instance
I20260812 05:23:14.700721 32144 external_mini_cluster.cc:1428] Setting key b6caa1a6e14d303cf448a9628a8d7700
Aug 12 05:23:14 dist-test-slave-42z9 krb5kdc[25677](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512194, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.100.3@KRBTEST.COM for kudu/127.31.100.62@KRBTEST.COM
I20260812 05:23:14.706084 25834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:23:14.698286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.100.3:59327 (local address 127.31.100.62:37129)
0812 05:23:14.698326 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0812 05:23:14.698328 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 05:23:14.698528 (+ 200us) server_negotiation.cc:407] Connection header received
0812 05:23:14.698973 (+ 445us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:23:14.698974 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:23:14.698985 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:23:14.699035 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:23:14.699906 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:23:14.700491 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:23:14.701702 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:23:14.701892 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:23:14.701945 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:23:14.703908 (+ 1963us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 05:23:14.703910 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 05:23:14.703910 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 05:23:14.703916 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0812 05:23:14.705002 (+ 1086us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:23:14.705338 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:23:14.705339 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:23:14.705339 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:23:14.705409 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:23:14.705627 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:23:14.705627 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:23:14.705627 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:23:14.705727 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 05:23:14.705775 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0812 05:23:14.705965 (+ 190us) server_negotiation.cc:299] Negotiation successful
0812 05:23:14.706021 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260812 05:23:14.706261 26583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:23:14.698270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.62:37129 (local address 127.31.100.3:59327)
0812 05:23:14.698403 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0812 05:23:14.698437 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0812 05:23:14.698899 (+ 462us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:23:14.699085 (+ 186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:23:14.699088 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:23:14.699645 (+ 557us) client_negotiation.cc:189] Negotiated authn=SASL
0812 05:23:14.699829 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:23:14.699834 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:23:14.700587 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:23:14.700588 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:23:14.701640 (+ 1052us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:23:14.701642 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:23:14.701951 (+ 309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:23:14.701951 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:23:14.702059 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:23:14.702490 (+ 431us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 05:23:14.702498 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0812 05:23:14.703811 (+ 1313us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 05:23:14.705073 (+ 1262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 05:23:14.705079 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 05:23:14.705080 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 05:23:14.705281 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 05:23:14.705480 (+ 199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 05:23:14.705481 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 05:23:14.705481 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 05:23:14.705577 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 05:23:14.705772 (+ 195us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 05:23:14.705774 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 05:23:14.705874 (+ 100us) client_negotiation.cc:769] Sending connection context
0812 05:23:14.705947 (+ 73us) client_negotiation.cc:240] Negotiation successful
0812 05:23:14.706072 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":68,"threads_started":1}
I20260812 05:23:14.706709 26580 heartbeater.cc:347] Connected to a master server at 127.31.100.62:37129
I20260812 05:23:14.706789 26580 heartbeater.cc:464] Registering TS with master...
I20260812 05:23:14.706924 26580 heartbeater.cc:511] Master 127.31.100.62:37129 requested a full tablet report, sending...
I20260812 05:23:14.707227 25762 ts_manager.cc:194] Registered new tserver with Master: df8db312fb6c4c57aa51f490c25fa09f (127.31.100.3:45871)
I20260812 05:23:14.707568 25762 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.100.3@KRBTEST.COM'} at 127.31.100.3:59327
I20260812 05:23:14.712319 32144 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 05:23:14.718762 25834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:23:14.713561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36012 (local address 127.31.100.62:37129)
0812 05:23:14.713606 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0812 05:23:14.713608 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 05:23:14.713731 (+ 123us) server_negotiation.cc:407] Connection header received
0812 05:23:14.713784 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:23:14.713785 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:23:14.713795 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:23:14.713838 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:23:14.714341 (+ 503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:23:14.714817 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:23:14.715513 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:23:14.715707 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:23:14.715759 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:23:14.716743 (+ 984us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 05:23:14.716744 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 05:23:14.716745 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 05:23:14.716751 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0812 05:23:14.717760 (+ 1009us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:23:14.718060 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:23:14.718061 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:23:14.718061 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:23:14.718112 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:23:14.718269 (+ 157us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:23:14.718270 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:23:14.718270 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:23:14.718413 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 05:23:14.718500 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0812 05:23:14.718609 (+ 109us) server_negotiation.cc:299] Negotiation successful
0812 05:23:14.718680 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260812 05:23:14.719945 25762 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36012:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 05:23:14.721474 25762 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 05:23:14.727222 26597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:23:14.723378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.1:37667 (local address 127.0.0.1:48590)
0812 05:23:14.723827 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0812 05:23:14.723837 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0812 05:23:14.723917 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:23:14.724065 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:23:14.724066 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:23:14.724068 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 05:23:14.724113 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:23:14.724114 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:23:14.724971 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:23:14.724972 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:23:14.726209 (+ 1237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:23:14.726212 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:23:14.726934 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:23:14.726935 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:23:14.727015 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:23:14.727029 (+ 14us) client_negotiation.cc:769] Sending connection context
0812 05:23:14.727101 (+ 72us) client_negotiation.cc:240] Negotiation successful
0812 05:23:14.727153 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":434,"thread_start_us":64,"threads_started":1}
I20260812 05:23:14.727231 26599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:23:14.723473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48590 (local address 127.31.100.1:37667)
0812 05:23:14.723627 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0812 05:23:14.723632 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 05:23:14.723929 (+ 297us) server_negotiation.cc:407] Connection header received
0812 05:23:14.723974 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:23:14.723977 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:23:14.723991 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:23:14.724053 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 05:23:14.724165 (+ 112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:23:14.724867 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:23:14.726301 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:23:14.726859 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:23:14.726927 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:23:14.726980 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0812 05:23:14.727116 (+ 136us) server_negotiation.cc:299] Negotiation successful
0812 05:23:14.727153 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":77,"threads_started":1}
I20260812 05:23:14.727476 26604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:23:14.723376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.3:45871 (local address 127.0.0.1:40410)
0812 05:23:14.723662 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0812 05:23:14.723699 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0812 05:23:14.723751 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:23:14.723989 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:23:14.723992 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:23:14.723994 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 05:23:14.724059 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:23:14.724060 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:23:14.725276 (+ 1216us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:23:14.725277 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:23:14.726352 (+ 1075us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:23:14.726354 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:23:14.727252 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:23:14.727253 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:23:14.727329 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:23:14.727332 (+ 3us) client_negotiation.cc:769] Sending connection context
0812 05:23:14.727375 (+ 43us) client_negotiation.cc:240] Negotiation successful
0812 05:23:14.727413 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":62,"threads_started":1}
I20260812 05:23:14.727682 26606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:23:14.723673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40410 (local address 127.31.100.3:45871)
0812 05:23:14.723832 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0812 05:23:14.723836 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 05:23:14.723844 (+ 8us) server_negotiation.cc:407] Connection header received
0812 05:23:14.723875 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:23:14.723877 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:23:14.723890 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:23:14.723967 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 05:23:14.724136 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:23:14.725195 (+ 1059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:23:14.726426 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:23:14.727111 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:23:14.727167 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:23:14.727218 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0812 05:23:14.727570 (+ 352us) server_negotiation.cc:299] Negotiation successful
0812 05:23:14.727611 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":83,"threads_started":1}
I20260812 05:23:14.728068 26598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:23:14.723568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.2:39191 (local address 127.0.0.1:39098)
0812 05:23:14.723978 (+ 410us) negotiation.cc:107] Waiting for socket to connect
0812 05:23:14.723986 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0812 05:23:14.724027 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:23:14.724606 (+ 579us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:23:14.724607 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:23:14.724609 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 05:23:14.724650 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:23:14.724652 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:23:14.725759 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:23:14.725760 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:23:14.726719 (+ 959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:23:14.726722 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:23:14.727837 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:23:14.727838 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:23:14.727910 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:23:14.727914 (+ 4us) client_negotiation.cc:769] Sending connection context
0812 05:23:14.727965 (+ 51us) client_negotiation.cc:240] Negotiation successful
0812 05:23:14.728002 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":401,"thread_start_us":49,"threads_started":1}
I20260812 05:23:14.728092 26609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:23:14.723693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39098 (local address 127.31.100.2:39191)
0812 05:23:14.723867 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0812 05:23:14.723871 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 05:23:14.724399 (+ 528us) server_negotiation.cc:407] Connection header received
0812 05:23:14.724447 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:23:14.724449 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:23:14.724463 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:23:14.724531 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 05:23:14.724718 (+ 187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:23:14.725677 (+ 959us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:23:14.726804 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:23:14.727702 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:23:14.727820 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:23:14.727873 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0812 05:23:14.727984 (+ 111us) server_negotiation.cc:299] Negotiation successful
0812 05:23:14.728027 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":72,"threads_started":1}
I20260812 05:23:14.728025 25988 tablet_service.cc:1467] Processing CreateTablet for tablet c3ca48be76d94bf9b30d01bd98e06e3e (DEFAULT_TABLE table=test-table [id=efa7108ddfc249af8fbe44e2d7d38c5e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 05:23:14.728371 25988 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c3ca48be76d94bf9b30d01bd98e06e3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 05:23:14.728284 26464 tablet_service.cc:1467] Processing CreateTablet for tablet c3ca48be76d94bf9b30d01bd98e06e3e (DEFAULT_TABLE table=test-table [id=efa7108ddfc249af8fbe44e2d7d38c5e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 05:23:14.728572 26464 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c3ca48be76d94bf9b30d01bd98e06e3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 05:23:14.728752 26235 tablet_service.cc:1467] Processing CreateTablet for tablet c3ca48be76d94bf9b30d01bd98e06e3e (DEFAULT_TABLE table=test-table [id=efa7108ddfc249af8fbe44e2d7d38c5e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 05:23:14.729063 26235 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c3ca48be76d94bf9b30d01bd98e06e3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 05:23:14.731730 26612 tablet_bootstrap.cc:492] T c3ca48be76d94bf9b30d01bd98e06e3e P df8db312fb6c4c57aa51f490c25fa09f: Bootstrap starting.
I20260812 05:23:14.731969 26613 tablet_bootstrap.cc:492] T c3ca48be76d94bf9b30d01bd98e06e3e P 3f75ed97c1aa4752b59f6a1551419390: Bootstrap starting.
I20260812 05:23:14.732301 26612 tablet_bootstrap.cc:654] T c3ca48be76d94bf9b30d01bd98e06e3e P df8db312fb6c4c57aa51f490c25fa09f: Neither blocks nor log segments found. Creating new log.
I20260812 05:23:14.732488 26613 tablet_bootstrap.cc:654] T c3ca48be76d94bf9b30d01bd98e06e3e P 3f75ed97c1aa4752b59f6a1551419390: Neither blocks nor log segments found. Creating new log.
I20260812 05:23:14.732764 26612 log.cc:824] T c3ca48be76d94bf9b30d01bd98e06e3e P df8db312fb6c4c57aa51f490c25fa09f: Log is configured to *not* fsync() on all Append() calls
I20260812 05:23:14.732944 26613 log.cc:824] T c3ca48be76d94bf9b30d01bd98e06e3e P 3f75ed97c1aa4752b59f6a1551419390: Log is configured to *not* fsync() on all Append() calls
I20260812 05:23:14.733275 26610 tablet_bootstrap.cc:492] T c3ca48be76d94bf9b30d01bd98e06e3e P edf25888a7e648348650523730de9401: Bootstrap starting.
I20260812 05:23:14.733572 26612 tablet_bootstrap.cc:492] T c3ca48be76d94bf9b30d01bd98e06e3e P df8db312fb6c4c57aa51f490c25fa09f: No bootstrap required, opened a new log
I20260812 05:23:14.733631 26612 ts_tablet_manager.cc:1397] T c3ca48be76d94bf9b30d01bd98e06e3e P df8db312fb6c4c57aa51f490c25fa09f: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260812 05:23:14.733737 26613 tablet_bootstrap.cc:492] T c3ca48be76d94bf9b30d01bd98e06e3e P 3f75ed97c1aa4752b59f6a1551419390: No bootstrap required, opened a new log
I20260812 05:23:14.733788 26613 ts_tablet_manager.cc:1397] T c3ca48be76d94bf9b30d01bd98e06e3e P 3f75ed97c1aa4752b59f6a1551419390: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260812 05:23:14.735141 26612 raft_consensus.cc:348] T c3ca48be76d94bf9b30d01bd98e06e3e P df8db312fb6c4c57aa51f490c25fa09f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df8db312fb6c4c57aa51f490c25fa09f" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 45871 } } peers { permanent_uuid: "edf25888a7e648348650523730de9401" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 39191 } } peers { permanent_uuid: "3f75ed97c1aa4752b59f6a1551419390" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 37667 } }
I20260812 05:23:14.735180 26613 raft_consensus.cc:348] T c3ca48be76d94bf9b30d01bd98e06e3e P 3f75ed97c1aa4752b59f6a1551419390 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df8db312fb6c4c57aa51f490c25fa09f" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 45871 } } peers { permanent_uuid: "edf25888a7e648348650523730de9401" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 39191 } } peers { permanent_uuid: "3f75ed97c1aa4752b59f6a1551419390" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 37667 } }
I20260812 05:23:14.735280 26612 raft_consensus.cc:374] T c3ca48be76d94bf9b30d01bd98e06e3e P df8db312fb6c4c57aa51f490c25fa09f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 05:23:14.735296 26613 raft_consensus.cc:374] T c3ca48be76d94bf9b30d01bd98e06e3e P 3f75ed97c1aa4752b59f6a1551419390 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 05:23:14.735316 26613 raft_consensus.cc:729] T c3ca48be76d94bf9b30d01bd98e06e3e P 3f75ed97c1aa4752b59f6a1551419390 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f75ed97c1aa4752b59f6a1551419390, State: Initialized, Role: FOLLOWER
I20260812 05:23:14.735312 26612 raft_consensus.cc:729] T c3ca48be76d94bf9b30d01bd98e06e3e P df8db312fb6c4c57aa51f490c25fa09f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df8db312fb6c4c57aa51f490c25fa09f, State: Initialized, Role: FOLLOWER
I20260812 05:23:14.735409 26613 consensus_queue.cc:260] T c3ca48be76d94bf9b30d01bd98e06e3e P 3f75ed97c1aa4752b59f6a1551419390 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df8db312fb6c4c57aa51f490c25fa09f" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 45871 } } peers { permanent_uuid: "edf25888a7e648348650523730de9401" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 39191 } } peers { permanent_uuid: "3f75ed97c1aa4752b59f6a1551419390" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 37667 } }
I20260812 05:23:14.735462 26612 consensus_queue.cc:260] T c3ca48be76d94bf9b30d01bd98e06e3e P df8db312fb6c4c57aa51f490c25fa09f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df8db312fb6c4c57aa51f490c25fa09f" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 45871 } } peers { permanent_uuid: "edf25888a7e648348650523730de9401" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 39191 } } peers { permanent_uuid: "3f75ed97c1aa4752b59f6a1551419390" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 37667 } }
I20260812 05:23:14.735697 26112 heartbeater.cc:503] Master 127.31.100.62:37129 was elected leader, sending a full tablet report...
I20260812 05:23:14.735750 26612 ts_tablet_manager.cc:1428] T c3ca48be76d94bf9b30d01bd98e06e3e P df8db312fb6c4c57aa51f490c25fa09f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260812 05:23:14.735929 26580 heartbeater.cc:503] Master 127.31.100.62:37129 was elected leader, sending a full tablet report...
I20260812 05:23:14.736147 26613 ts_tablet_manager.cc:1428] T c3ca48be76d94bf9b30d01bd98e06e3e P 3f75ed97c1aa4752b59f6a1551419390: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260812 05:23:14.736668 26610 tablet_bootstrap.cc:654] T c3ca48be76d94bf9b30d01bd98e06e3e P edf25888a7e648348650523730de9401: Neither blocks nor log segments found. Creating new log.
I20260812 05:23:14.737164 26610 log.cc:824] T c3ca48be76d94bf9b30d01bd98e06e3e P edf25888a7e648348650523730de9401: Log is configured to *not* fsync() on all Append() calls
I20260812 05:23:14.737818 26610 tablet_bootstrap.cc:492] T c3ca48be76d94bf9b30d01bd98e06e3e P edf25888a7e648348650523730de9401: No bootstrap required, opened a new log
I20260812 05:23:14.737869 26610 ts_tablet_manager.cc:1397] T c3ca48be76d94bf9b30d01bd98e06e3e P edf25888a7e648348650523730de9401: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20260812 05:23:14.739221 26610 raft_consensus.cc:348] T c3ca48be76d94bf9b30d01bd98e06e3e P edf25888a7e648348650523730de9401 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df8db312fb6c4c57aa51f490c25fa09f" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 45871 } } peers { permanent_uuid: "edf25888a7e648348650523730de9401" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 39191 } } peers { permanent_uuid: "3f75ed97c1aa4752b59f6a1551419390" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 37667 } }
I20260812 05:23:14.739352 26610 raft_consensus.cc:374] T c3ca48be76d94bf9b30d01bd98e06e3e P edf25888a7e648348650523730de9401 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 05:23:14.739390 26610 raft_consensus.cc:729] T c3ca48be76d94bf9b30d01bd98e06e3e P edf25888a7e648348650523730de9401 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: edf25888a7e648348650523730de9401, State: Initialized, Role: FOLLOWER
I20260812 05:23:14.739492 26610 consensus_queue.cc:260] T c3ca48be76d94bf9b30d01bd98e06e3e P edf25888a7e648348650523730de9401 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df8db312fb6c4c57aa51f490c25fa09f" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 45871 } } peers { permanent_uuid: "edf25888a7e648348650523730de9401" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 39191 } } peers { permanent_uuid: "3f75ed97c1aa4752b59f6a1551419390" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 37667 } }
I20260812 05:23:14.739717 26610 ts_tablet_manager.cc:1428] T c3ca48be76d94bf9b30d01bd98e06e3e P edf25888a7e648348650523730de9401: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260812 05:23:14.739749 26354 heartbeater.cc:503] Master 127.31.100.62:37129 was elected leader, sending a full tablet report...
W20260812 05:23:14.797997 26355 tablet.cc:2369] T c3ca48be76d94bf9b30d01bd98e06e3e P edf25888a7e648348650523730de9401: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 05:23:14.894688 26115 tablet.cc:2369] T c3ca48be76d94bf9b30d01bd98e06e3e P 3f75ed97c1aa4752b59f6a1551419390: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 05:23:14.948202 26582 tablet.cc:2369] T c3ca48be76d94bf9b30d01bd98e06e3e P df8db312fb6c4c57aa51f490c25fa09f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 05:23:15.125641 26617 raft_consensus.cc:482] T c3ca48be76d94bf9b30d01bd98e06e3e P df8db312fb6c4c57aa51f490c25fa09f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 05:23:15.125771 26617 raft_consensus.cc:504] T c3ca48be76d94bf9b30d01bd98e06e3e P df8db312fb6c4c57aa51f490c25fa09f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df8db312fb6c4c57aa51f490c25fa09f" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 45871 } } peers { permanent_uuid: "edf25888a7e648348650523730de9401" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 39191 } } peers { permanent_uuid: "3f75ed97c1aa4752b59f6a1551419390" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 37667 } }
I20260812 05:23:15.126153 26617 leader_election.cc:288] T c3ca48be76d94bf9b30d01bd98e06e3e P df8db312fb6c4c57aa51f490c25fa09f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers edf25888a7e648348650523730de9401 (127.31.100.2:39191), 3f75ed97c1aa4752b59f6a1551419390 (127.31.100.1:37667)
I20260812 05:23:15.128754 26599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:23:15.126439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.100.3:42537 (local address 127.31.100.1:37667)
0812 05:23:15.126482 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0812 05:23:15.126484 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 05:23:15.126495 (+ 11us) server_negotiation.cc:407] Connection header received
0812 05:23:15.126566 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:23:15.126567 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:23:15.126575 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:23:15.126620 (+ 45us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 05:23:15.126758 (+ 138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:23:15.127234 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:23:15.128109 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:23:15.128454 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:23:15.128516 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:23:15.128533 (+ 17us) server_negotiation.cc:1035] Waiting for connection context
0812 05:23:15.128658 (+ 125us) server_negotiation.cc:299] Negotiation successful
0812 05:23:15.128694 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260812 05:23:15.128754 26583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:23:15.126357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.1:37667 (local address 127.31.100.3:42537)
0812 05:23:15.126420 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0812 05:23:15.126432 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0812 05:23:15.126485 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:23:15.126628 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:23:15.126629 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:23:15.126631 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 05:23:15.126700 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:23:15.126702 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:23:15.127301 (+ 599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:23:15.127302 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:23:15.128013 (+ 711us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:23:15.128015 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:23:15.128518 (+ 503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:23:15.128518 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:23:15.128595 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:23:15.128598 (+ 3us) client_negotiation.cc:769] Sending connection context
0812 05:23:15.128643 (+ 45us) client_negotiation.cc:240] Negotiation successful
0812 05:23:15.128680 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260812 05:23:15.128844 26609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:23:15.126451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.100.3:53675 (local address 127.31.100.2:39191)
0812 05:23:15.126497 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0812 05:23:15.126499 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 05:23:15.126617 (+ 118us) server_negotiation.cc:407] Connection header received
0812 05:23:15.126678 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:23:15.126679 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:23:15.126688 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:23:15.126734 (+ 46us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 05:23:15.126866 (+ 132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:23:15.127312 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:23:15.128193 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:23:15.128538 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:23:15.128601 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:23:15.128610 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0812 05:23:15.128741 (+ 131us) server_negotiation.cc:299] Negotiation successful
0812 05:23:15.128774 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260812 05:23:15.128875 26623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:23:15.126365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.2:39191 (local address 127.31.100.3:53675)
0812 05:23:15.126550 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0812 05:23:15.126560 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0812 05:23:15.126617 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:23:15.126740 (+ 123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:23:15.126741 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:23:15.126743 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 05:23:15.126811 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:23:15.126812 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:23:15.127393 (+ 581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:23:15.127394 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:23:15.128113 (+ 719us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:23:15.128115 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:23:15.128606 (+ 491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:23:15.128607 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:23:15.128682 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:23:15.128686 (+ 4us) client_negotiation.cc:769] Sending connection context
0812 05:23:15.128730 (+ 44us) client_negotiation.cc:240] Negotiation successful
0812 05:23:15.128766 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":170,"mutex_wait_us":15,"thread_start_us":95,"threads_started":1}
I20260812 05:23:15.129096 26023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3ca48be76d94bf9b30d01bd98e06e3e" candidate_uuid: "df8db312fb6c4c57aa51f490c25fa09f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f75ed97c1aa4752b59f6a1551419390" is_pre_election: true
I20260812 05:23:15.129221 26023 raft_consensus.cc:2454] T c3ca48be76d94bf9b30d01bd98e06e3e P 3f75ed97c1aa4752b59f6a1551419390 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df8db312fb6c4c57aa51f490c25fa09f in term 0.
I20260812 05:23:15.129474 26391 leader_election.cc:302] T c3ca48be76d94bf9b30d01bd98e06e3e P df8db312fb6c4c57aa51f490c25fa09f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f75ed97c1aa4752b59f6a1551419390, df8db312fb6c4c57aa51f490c25fa09f; no voters:
I20260812 05:23:15.129508 26265 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3ca48be76d94bf9b30d01bd98e06e3e" candidate_uuid: "df8db312fb6c4c57aa51f490c25fa09f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "edf25888a7e648348650523730de9401" is_pre_election: true
I20260812 05:23:15.129649 26265 raft_consensus.cc:2454] T c3ca48be76d94bf9b30d01bd98e06e3e P edf25888a7e648348650523730de9401 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df8db312fb6c4c57aa51f490c25fa09f in term 0.
I20260812 05:23:15.129648 26617 raft_consensus.cc:2781] T c3ca48be76d94bf9b30d01bd98e06e3e P df8db312fb6c4c57aa51f490c25fa09f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 05:23:15.129726 26617 raft_consensus.cc:482] T c3ca48be76d94bf9b30d01bd98e06e3e P df8db312fb6c4c57aa51f490c25fa09f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 05:23:15.129753 26617 raft_consensus.cc:3037] T c3ca48be76d94bf9b30d01bd98e06e3e P df8db312fb6c4c57aa51f490c25fa09f [term 0 FOLLOWER]: Advancing to term 1
I20260812 05:23:15.130465 26617 raft_consensus.cc:504] T c3ca48be76d94bf9b30d01bd98e06e3e P df8db312fb6c4c57aa51f490c25fa09f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df8db312fb6c4c57aa51f490c25fa09f" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 45871 } } peers { permanent_uuid: "edf25888a7e648348650523730de9401" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 39191 } } peers { permanent_uuid: "3f75ed97c1aa4752b59f6a1551419390" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 37667 } }
I20260812 05:23:15.130579 26617 leader_election.cc:288] T c3ca48be76d94bf9b30d01bd98e06e3e P df8db312fb6c4c57aa51f490c25fa09f [CANDIDATE]: Term 1 election: Requested vote from peers edf25888a7e648348650523730de9401 (127.31.100.2:39191), 3f75ed97c1aa4752b59f6a1551419390 (127.31.100.1:37667)
I20260812 05:23:15.130716 26023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3ca48be76d94bf9b30d01bd98e06e3e" candidate_uuid: "df8db312fb6c4c57aa51f490c25fa09f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f75ed97c1aa4752b59f6a1551419390"
I20260812 05:23:15.130766 26265 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3ca48be76d94bf9b30d01bd98e06e3e" candidate_uuid: "df8db312fb6c4c57aa51f490c25fa09f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "edf25888a7e648348650523730de9401"
I20260812 05:23:15.130798 26023 raft_consensus.cc:3037] T c3ca48be76d94bf9b30d01bd98e06e3e P 3f75ed97c1aa4752b59f6a1551419390 [term 0 FOLLOWER]: Advancing to term 1
I20260812 05:23:15.130828 26265 raft_consensus.cc:3037] T c3ca48be76d94bf9b30d01bd98e06e3e P edf25888a7e648348650523730de9401 [term 0 FOLLOWER]: Advancing to term 1
I20260812 05:23:15.131492 26023 raft_consensus.cc:2454] T c3ca48be76d94bf9b30d01bd98e06e3e P 3f75ed97c1aa4752b59f6a1551419390 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df8db312fb6c4c57aa51f490c25fa09f in term 1.
I20260812 05:23:15.131500 26265 raft_consensus.cc:2454] T c3ca48be76d94bf9b30d01bd98e06e3e P edf25888a7e648348650523730de9401 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df8db312fb6c4c57aa51f490c25fa09f in term 1.
I20260812 05:23:15.131669 26391 leader_election.cc:302] T c3ca48be76d94bf9b30d01bd98e06e3e P df8db312fb6c4c57aa51f490c25fa09f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f75ed97c1aa4752b59f6a1551419390, df8db312fb6c4c57aa51f490c25fa09f; no voters:
I20260812 05:23:15.131742 26617 raft_consensus.cc:2781] T c3ca48be76d94bf9b30d01bd98e06e3e P df8db312fb6c4c57aa51f490c25fa09f [term 1 FOLLOWER]: Leader election won for term 1
I20260812 05:23:15.131867 26617 raft_consensus.cc:686] T c3ca48be76d94bf9b30d01bd98e06e3e P df8db312fb6c4c57aa51f490c25fa09f [term 1 LEADER]: Becoming Leader. State: Replica: df8db312fb6c4c57aa51f490c25fa09f, State: Running, Role: LEADER
I20260812 05:23:15.131963 26617 consensus_queue.cc:237] T c3ca48be76d94bf9b30d01bd98e06e3e P df8db312fb6c4c57aa51f490c25fa09f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df8db312fb6c4c57aa51f490c25fa09f" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 45871 } } peers { permanent_uuid: "edf25888a7e648348650523730de9401" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 39191 } } peers { permanent_uuid: "3f75ed97c1aa4752b59f6a1551419390" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 37667 } }
I20260812 05:23:15.132742 25761 catalog_manager.cc:5495] T c3ca48be76d94bf9b30d01bd98e06e3e P df8db312fb6c4c57aa51f490c25fa09f reported cstate change: term changed from 0 to 1, leader changed from <none> to df8db312fb6c4c57aa51f490c25fa09f (127.31.100.3). New cstate: current_term: 1 leader_uuid: "df8db312fb6c4c57aa51f490c25fa09f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df8db312fb6c4c57aa51f490c25fa09f" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 45871 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "edf25888a7e648348650523730de9401" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 39191 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3f75ed97c1aa4752b59f6a1551419390" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 37667 } health_report { overall_health: UNKNOWN } } }
I20260812 05:23:15.165148 26606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:23:15.162877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40420 (local address 127.31.100.3:45871)
0812 05:23:15.162946 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0812 05:23:15.162948 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 05:23:15.162956 (+ 8us) server_negotiation.cc:407] Connection header received
0812 05:23:15.162998 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:23:15.162999 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:23:15.163018 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:23:15.163077 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 05:23:15.163211 (+ 134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:23:15.163685 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:23:15.164408 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:23:15.164612 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:23:15.164667 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:23:15.164827 (+ 160us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 05:23:15.164937 (+ 110us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 05:23:15.165005 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0812 05:23:15.165028 (+ 23us) server_negotiation.cc:299] Negotiation successful
0812 05:23:15.165051 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260812 05:23:15.166996 26023 raft_consensus.cc:1261] T c3ca48be76d94bf9b30d01bd98e06e3e P 3f75ed97c1aa4752b59f6a1551419390 [term 1 FOLLOWER]: Refusing update from remote peer df8db312fb6c4c57aa51f490c25fa09f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 05:23:15.166996 26265 raft_consensus.cc:1261] T c3ca48be76d94bf9b30d01bd98e06e3e P edf25888a7e648348650523730de9401 [term 1 FOLLOWER]: Refusing update from remote peer df8db312fb6c4c57aa51f490c25fa09f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 05:23:15.167338 26625 consensus_queue.cc:1035] T c3ca48be76d94bf9b30d01bd98e06e3e P df8db312fb6c4c57aa51f490c25fa09f [LEADER]: Connected to new peer: Peer: permanent_uuid: "edf25888a7e648348650523730de9401" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 39191 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 05:23:15.167486 26617 consensus_queue.cc:1035] T c3ca48be76d94bf9b30d01bd98e06e3e P df8db312fb6c4c57aa51f490c25fa09f [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f75ed97c1aa4752b59f6a1551419390" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 37667 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 05:23:15.169731 26629 mvcc.cc:205] Tried to move back new op lower bound from 7317553951401201664 to 7317553951260975104. Current Snapshot: MvccSnapshot[applied={T|T < 7317553951401201664}]
I20260812 05:23:15.170311 26630 mvcc.cc:205] Tried to move back new op lower bound from 7317553951401201664 to 7317553951260975104. Current Snapshot: MvccSnapshot[applied={T|T < 7317553951401201664}]
I20260812 05:23:15.172539 25759 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36012:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 05:23:15.172640 25759 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36012:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 05:23:15.173348 25759 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a2ede0bc8af24780a89103af190035b8: Sending DeleteTablet for 3 replicas of tablet c3ca48be76d94bf9b30d01bd98e06e3e
I20260812 05:23:15.173713 26235 tablet_service.cc:1514] Processing DeleteTablet for tablet c3ca48be76d94bf9b30d01bd98e06e3e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 05:23:15 UTC) from {username='kudu', principal='kudu/127.31.100.62@KRBTEST.COM'} at 127.0.0.1:39098
I20260812 05:23:15.173712 25988 tablet_service.cc:1514] Processing DeleteTablet for tablet c3ca48be76d94bf9b30d01bd98e06e3e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 05:23:15 UTC) from {username='kudu', principal='kudu/127.31.100.62@KRBTEST.COM'} at 127.0.0.1:48590
I20260812 05:23:15.173924 26464 tablet_service.cc:1514] Processing DeleteTablet for tablet c3ca48be76d94bf9b30d01bd98e06e3e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 05:23:15 UTC) from {username='kudu', principal='kudu/127.31.100.62@KRBTEST.COM'} at 127.0.0.1:40410
I20260812 05:23:15.173980 32144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0waFvT/build/release/bin/kudu with pid 25876
I20260812 05:23:15.174129 26651 tablet_replica.cc:330] T c3ca48be76d94bf9b30d01bd98e06e3e P df8db312fb6c4c57aa51f490c25fa09f: stopping tablet replica
I20260812 05:23:15.174263 26651 raft_consensus.cc:2229] T c3ca48be76d94bf9b30d01bd98e06e3e P df8db312fb6c4c57aa51f490c25fa09f [term 1 LEADER]: Raft consensus shutting down.
I20260812 05:23:15.174465 26646 tablet_replica.cc:330] T c3ca48be76d94bf9b30d01bd98e06e3e P edf25888a7e648348650523730de9401: stopping tablet replica
I20260812 05:23:15.174667 26646 raft_consensus.cc:2229] T c3ca48be76d94bf9b30d01bd98e06e3e P edf25888a7e648348650523730de9401 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 05:23:15.176770 26646 raft_consensus.cc:2258] T c3ca48be76d94bf9b30d01bd98e06e3e P edf25888a7e648348650523730de9401 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 05:23:15.179728 26651 raft_consensus.cc:2258] T c3ca48be76d94bf9b30d01bd98e06e3e P df8db312fb6c4c57aa51f490c25fa09f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 05:23:15.182813 26651 ts_tablet_manager.cc:1905] T c3ca48be76d94bf9b30d01bd98e06e3e P df8db312fb6c4c57aa51f490c25fa09f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 05:23:15.183692 26646 ts_tablet_manager.cc:1905] T c3ca48be76d94bf9b30d01bd98e06e3e P edf25888a7e648348650523730de9401: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 05:23:15.184649 26651 ts_tablet_manager.cc:1918] T c3ca48be76d94bf9b30d01bd98e06e3e P df8db312fb6c4c57aa51f490c25fa09f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 05:23:15.184804 26651 log.cc:1196] T c3ca48be76d94bf9b30d01bd98e06e3e P df8db312fb6c4c57aa51f490c25fa09f: Deleting WAL directory at /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786512017613366-32144-0/minicluster-data/ts-2/wal/wals/c3ca48be76d94bf9b30d01bd98e06e3e
I20260812 05:23:15.185186 26651 ts_tablet_manager.cc:1939] T c3ca48be76d94bf9b30d01bd98e06e3e P df8db312fb6c4c57aa51f490c25fa09f: Deleting consensus metadata
I20260812 05:23:15.185726 26646 ts_tablet_manager.cc:1918] T c3ca48be76d94bf9b30d01bd98e06e3e P edf25888a7e648348650523730de9401: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 05:23:15.185734 25737 catalog_manager.cc:4841] TS df8db312fb6c4c57aa51f490c25fa09f (127.31.100.3:45871): tablet c3ca48be76d94bf9b30d01bd98e06e3e (table test-table [id=efa7108ddfc249af8fbe44e2d7d38c5e]) successfully deleted
I20260812 05:23:15.185895 26646 log.cc:1196] T c3ca48be76d94bf9b30d01bd98e06e3e P edf25888a7e648348650523730de9401: Deleting WAL directory at /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786512017613366-32144-0/minicluster-data/ts-1/wal/wals/c3ca48be76d94bf9b30d01bd98e06e3e
I20260812 05:23:15.186219 26646 ts_tablet_manager.cc:1939] T c3ca48be76d94bf9b30d01bd98e06e3e P edf25888a7e648348650523730de9401: Deleting consensus metadata
I20260812 05:23:15.186707 25737 catalog_manager.cc:4841] TS edf25888a7e648348650523730de9401 (127.31.100.2:39191): tablet c3ca48be76d94bf9b30d01bd98e06e3e (table test-table [id=efa7108ddfc249af8fbe44e2d7d38c5e]) successfully deleted
W20260812 05:23:15.193235 25743 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.31.100.1:37667 (error 108)
I20260812 05:23:15.193495 32144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0waFvT/build/release/bin/kudu with pid 26121
I20260812 05:23:15.195804 26598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:23:15.195672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.1:37667 (local address 127.0.0.1:48604)
0812 05:23:15.195726 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0812 05:23:15.195768 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.100.1:37667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260812 05:23:15.196058 25743 catalog_manager.cc:4584] TS 3f75ed97c1aa4752b59f6a1551419390 (127.31.100.1:37667): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c3ca48be76d94bf9b30d01bd98e06e3e: Network error: Client connection negotiation failed: client connection to 127.31.100.1:37667: connect: Connection refused (error 111)
I20260812 05:23:15.204149 32144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0waFvT/build/release/bin/kudu with pid 26360
I20260812 05:23:15.214167 32144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0waFvT/build/release/bin/kudu with pid 25699
I20260812 05:23:35.222992 32144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0waFvT/build/release/bin/kudu
/tmp/dist-test-task0waFvT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786512017613366-32144-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786512017613366-32144-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786512017613366-32144-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786512017613366-32144-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.100.62:37129
--webserver_interface=127.31.100.62
--webserver_port=43699
--keytab_file=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.62.keytab
--principal=kudu/127.31.100.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.100.20:45999
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.100.62:37129
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786512017613366-32144-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786512017613366-32144-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786512017613366-32144-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 05:23:35.305480 26654 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 05:23:35.305673 26654 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 05:23:35.305693 26654 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 05:23:35.306757 26654 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260812 05:23:35.306788 26654 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260812 05:23:35.306799 26654 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260812 05:23:35.306809 26654 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 05:23:35.306821 26654 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 05:23:35.306840 26654 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 05:23:35.306857 26654 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 05:23:35.306869 26654 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 05:23:35.308256 26654 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.100.20:45999
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786512017613366-32144-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786512017613366-32144-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.31.100.62:37129
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.62.keytab
--principal=kudu/127.31.100.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.100.62:37129
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786512017613366-32144-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.100.62
--webserver_port=43699
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26654
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786512017613366-32144-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 04:20:34 UTC on ccycloud.quasar-phqdqp.root.comops.site
build id 4581871
I20260812 05:23:35.308449 26654 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 05:23:35.308615 26654 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 05:23:35.317509 26667 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 05:23:35.317608 26662 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 05:23:35.317633 26669 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 05:23:35.317533 26654 server_base.cc:1034] running on GCE node
I20260812 05:23:35.317919 26654 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 05:23:35.318203 26654 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 05:23:35.319334 26654 hybrid_clock.cc:648] HybridClock initialized: now 1786512215319317 us; error 34 us; skew 500 ppm
Aug 12 05:23:35 dist-test-slave-42z9 krb5kdc[25677](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512215, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.100.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 05:23:35.324553 26654 init.cc:377] Logged in from keytab as kudu/127.31.100.62@KRBTEST.COM (short username kudu)
I20260812 05:23:35.324973 26654 webserver.cc:460] Webserver started at http://127.31.100.62:43699/ using document root <none> and password file <none>
I20260812 05:23:35.325187 26654 fs_manager.cc:359] Metadata directory not provided
I20260812 05:23:35.325242 26654 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 05:23:35.326589 26654 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260812 05:23:35.327200 26677 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 05:23:35.327382 26654 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260812 05:23:35.327474 26654 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786512017613366-32144-0/minicluster-data/master-0/data,/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786512017613366-32144-0/minicluster-data/master-0/wal
uuid: "a2ede0bc8af24780a89103af190035b8"
format_stamp: "Formatted at 2026-08-12 05:23:14 on dist-test-slave-42z9"
server_key: "a0b2f7fff55987d6788b7a426193d037"
server_key_iv: "bf518a542e138ca511cd0bbdc2024334"
server_key_version: "encryptionkey@0"
I20260812 05:23:35.327898 26654 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786512017613366-32144-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786512017613366-32144-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786512017613366-32144-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 05:23:35.339687 26654 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 05:23:35.341699 26654 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 05:23:35.341826 26654 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 05:23:35.345726 26654 rpc_server.cc:307] RPC server started. Bound to: 127.31.100.62:37129
I20260812 05:23:35.345763 26776 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.100.62:37129 every 8 connection(s)
I20260812 05:23:35.346136 26654 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786512017613366-32144-0/minicluster-data/master-0/data/info.pb
I20260812 05:23:35.347549 26777 sys_catalog.cc:263] Verifying existing consensus state
I20260812 05:23:35.348225 26777 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a2ede0bc8af24780a89103af190035b8: Bootstrap starting.
I20260812 05:23:35.348517 32144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0waFvT/build/release/bin/kudu as pid 26654
I20260812 05:23:35.348763 32144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0waFvT/build/release/bin/kudu
/tmp/dist-test-task0waFvT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786512017613366-32144-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786512017613366-32144-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786512017613366-32144-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786512017613366-32144-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.100.1:37667
--local_ip_for_outbound_sockets=127.31.100.1
--tserver_master_addrs=127.31.100.62:37129
--webserver_port=43255
--webserver_interface=127.31.100.1
--keytab_file=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.1.keytab
--principal=kudu/127.31.100.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.100.20:45999
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786512017613366-32144-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786512017613366-32144-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786512017613366-32144-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 05:23:35.356900 26777 log.cc:824] T 00000000000000000000000000000000 P a2ede0bc8af24780a89103af190035b8: Log is configured to *not* fsync() on all Append() calls
I20260812 05:23:35.358716 26777 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a2ede0bc8af24780a89103af190035b8: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 05:23:35.358992 26777 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a2ede0bc8af24780a89103af190035b8: Bootstrap complete.
I20260812 05:23:35.361070 26777 raft_consensus.cc:348] T 00000000000000000000000000000000 P a2ede0bc8af24780a89103af190035b8 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2ede0bc8af24780a89103af190035b8" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 37129 } }
I20260812 05:23:35.361284 26777 raft_consensus.cc:729] T 00000000000000000000000000000000 P a2ede0bc8af24780a89103af190035b8 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2ede0bc8af24780a89103af190035b8, State: Initialized, Role: FOLLOWER
I20260812 05:23:35.361421 26777 consensus_queue.cc:260] T 00000000000000000000000000000000 P a2ede0bc8af24780a89103af190035b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2ede0bc8af24780a89103af190035b8" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 37129 } }
I20260812 05:23:35.361515 26777 raft_consensus.cc:388] T 00000000000000000000000000000000 P a2ede0bc8af24780a89103af190035b8 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 05:23:35.361570 26777 raft_consensus.cc:482] T 00000000000000000000000000000000 P a2ede0bc8af24780a89103af190035b8 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 05:23:35.361617 26777 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a2ede0bc8af24780a89103af190035b8 [term 1 FOLLOWER]: Advancing to term 2
I20260812 05:23:35.362573 26777 raft_consensus.cc:504] T 00000000000000000000000000000000 P a2ede0bc8af24780a89103af190035b8 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2ede0bc8af24780a89103af190035b8" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 37129 } }
I20260812 05:23:35.362669 26777 leader_election.cc:302] T 00000000000000000000000000000000 P a2ede0bc8af24780a89103af190035b8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a2ede0bc8af24780a89103af190035b8; no voters:
I20260812 05:23:35.362818 26777 leader_election.cc:288] T 00000000000000000000000000000000 P a2ede0bc8af24780a89103af190035b8 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260812 05:23:35.362864 26784 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a2ede0bc8af24780a89103af190035b8 [term 2 FOLLOWER]: Leader election won for term 2
I20260812 05:23:35.363020 26784 raft_consensus.cc:686] T 00000000000000000000000000000000 P a2ede0bc8af24780a89103af190035b8 [term 2 LEADER]: Becoming Leader. State: Replica: a2ede0bc8af24780a89103af190035b8, State: Running, Role: LEADER
I20260812 05:23:35.363091 26777 sys_catalog.cc:564] T 00000000000000000000000000000000 P a2ede0bc8af24780a89103af190035b8 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 05:23:35.363106 26784 consensus_queue.cc:237] T 00000000000000000000000000000000 P a2ede0bc8af24780a89103af190035b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2ede0bc8af24780a89103af190035b8" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 37129 } }
I20260812 05:23:35.363482 26785 sys_catalog.cc:455] T 00000000000000000000000000000000 P a2ede0bc8af24780a89103af190035b8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "a2ede0bc8af24780a89103af190035b8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2ede0bc8af24780a89103af190035b8" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 37129 } } }
I20260812 05:23:35.363505 26784 sys_catalog.cc:455] T 00000000000000000000000000000000 P a2ede0bc8af24780a89103af190035b8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a2ede0bc8af24780a89103af190035b8. Latest consensus state: current_term: 2 leader_uuid: "a2ede0bc8af24780a89103af190035b8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2ede0bc8af24780a89103af190035b8" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 37129 } } }
I20260812 05:23:35.363566 26785 sys_catalog.cc:458] T 00000000000000000000000000000000 P a2ede0bc8af24780a89103af190035b8 [sys.catalog]: This master's current role is: LEADER
I20260812 05:23:35.363579 26784 sys_catalog.cc:458] T 00000000000000000000000000000000 P a2ede0bc8af24780a89103af190035b8 [sys.catalog]: This master's current role is: LEADER
I20260812 05:23:35.363827 26801 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 05:23:35.364663 26801 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 05:23:35.364833 26801 catalog_manager.cc:1259] Loaded cluster ID: 946abde4098a48769bf5aba02bbd3087
I20260812 05:23:35.365016 26801 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 05:23:35.365871 26801 catalog_manager.cc:1504] Loading token signing keys...
I20260812 05:23:35.366034 26801 catalog_manager.cc:5879] T 00000000000000000000000000000000 P a2ede0bc8af24780a89103af190035b8: Loaded TSK: 0
I20260812 05:23:35.379793 26801 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a2ede0bc8af24780a89103af190035b8: Generated new TSK 1
I20260812 05:23:35.379899 26801 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260812 05:23:35.448889 26779 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 05:23:35.449095 26779 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 05:23:35.449126 26779 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 05:23:35.450270 26779 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 05:23:35.450312 26779 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 05:23:35.450347 26779 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.100.1
I20260812 05:23:35.451777 26779 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.100.20:45999
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786512017613366-32144-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786512017613366-32144-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.1.keytab
--principal=kudu/127.31.100.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.100.1:37667
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786512017613366-32144-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.100.1
--webserver_port=43255
--tserver_master_addrs=127.31.100.62:37129
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26779
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.100.1
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786512017613366-32144-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 04:20:34 UTC on ccycloud.quasar-phqdqp.root.comops.site
build id 4581871
I20260812 05:23:35.452010 26779 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 05:23:35.452214 26779 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 05:23:35.457978 26824 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 05:23:35.458002 26822 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 05:23:35.458011 26827 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 05:23:35.458189 26779 server_base.cc:1034] running on GCE node
I20260812 05:23:35.458472 26779 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 05:23:35.458702 26779 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 05:23:35.459846 26779 hybrid_clock.cc:648] HybridClock initialized: now 1786512215459828 us; error 32 us; skew 500 ppm
Aug 12 05:23:35 dist-test-slave-42z9 krb5kdc[25677](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512215, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.100.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 05:23:35.462275 26779 init.cc:377] Logged in from keytab as kudu/127.31.100.1@KRBTEST.COM (short username kudu)
I20260812 05:23:35.462742 26779 webserver.cc:460] Webserver started at http://127.31.100.1:43255/ using document root <none> and password file <none>
I20260812 05:23:35.462958 26779 fs_manager.cc:359] Metadata directory not provided
I20260812 05:23:35.463021 26779 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 05:23:35.464383 26779 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260812 05:23:35.465029 26838 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 05:23:35.465193 26779 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260812 05:23:35.465301 26779 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786512017613366-32144-0/minicluster-data/ts-0/data,/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786512017613366-32144-0/minicluster-data/ts-0/wal
uuid: "3f75ed97c1aa4752b59f6a1551419390"
format_stamp: "Formatted at 2026-08-12 05:23:14 on dist-test-slave-42z9"
server_key: "1ff21222bd34bb43ebd41add3e2d3a43"
server_key_iv: "a68b8fa533bdffcbc4072c9da796609f"
server_key_version: "encryptionkey@0"
I20260812 05:23:35.465765 26779 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786512017613366-32144-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786512017613366-32144-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786512017613366-32144-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 05:23:35.481936 26779 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 05:23:35.484195 26779 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 05:23:35.484347 26779 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 05:23:35.484611 26779 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 05:23:35.485095 26853 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260812 05:23:35.486701 26779 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260812 05:23:35.486763 26779 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260812 05:23:35.486799 26779 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260812 05:23:35.487663 26779 ts_tablet_manager.cc:610] Registered 1 tablets
I20260812 05:23:35.487712 26779 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260812 05:23:35.487761 26853 tablet_bootstrap.cc:492] T c3ca48be76d94bf9b30d01bd98e06e3e P 3f75ed97c1aa4752b59f6a1551419390: Bootstrap starting.
I20260812 05:23:35.494849 26779 rpc_server.cc:307] RPC server started. Bound to: 127.31.100.1:37667
I20260812 05:23:35.495326 26779 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786512017613366-32144-0/minicluster-data/ts-0/data/info.pb
I20260812 05:23:35.497470 26853 log.cc:824] T c3ca48be76d94bf9b30d01bd98e06e3e P 3f75ed97c1aa4752b59f6a1551419390: Log is configured to *not* fsync() on all Append() calls
I20260812 05:23:35.497787 27049 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.100.1:37667 every 8 connection(s)
I20260812 05:23:35.500164 26853 tablet_bootstrap.cc:492] T c3ca48be76d94bf9b30d01bd98e06e3e P 3f75ed97c1aa4752b59f6a1551419390: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 05:23:35.500460 26853 tablet_bootstrap.cc:492] T c3ca48be76d94bf9b30d01bd98e06e3e P 3f75ed97c1aa4752b59f6a1551419390: Bootstrap complete.
I20260812 05:23:35.501426 26853 ts_tablet_manager.cc:1397] T c3ca48be76d94bf9b30d01bd98e06e3e P 3f75ed97c1aa4752b59f6a1551419390: Time spent bootstrapping tablet: real 0.014s user 0.001s sys 0.009s
I20260812 05:23:35.502518 26853 raft_consensus.cc:348] T c3ca48be76d94bf9b30d01bd98e06e3e P 3f75ed97c1aa4752b59f6a1551419390 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df8db312fb6c4c57aa51f490c25fa09f" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 45871 } } peers { permanent_uuid: "edf25888a7e648348650523730de9401" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 39191 } } peers { permanent_uuid: "3f75ed97c1aa4752b59f6a1551419390" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 37667 } }
I20260812 05:23:35.502827 26853 raft_consensus.cc:729] T c3ca48be76d94bf9b30d01bd98e06e3e P 3f75ed97c1aa4752b59f6a1551419390 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f75ed97c1aa4752b59f6a1551419390, State: Initialized, Role: FOLLOWER
I20260812 05:23:35.502960 26853 consensus_queue.cc:260] T c3ca48be76d94bf9b30d01bd98e06e3e P 3f75ed97c1aa4752b59f6a1551419390 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df8db312fb6c4c57aa51f490c25fa09f" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 45871 } } peers { permanent_uuid: "edf25888a7e648348650523730de9401" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 39191 } } peers { permanent_uuid: "3f75ed97c1aa4752b59f6a1551419390" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 37667 } }
I20260812 05:23:35.503257 26853 ts_tablet_manager.cc:1428] T c3ca48be76d94bf9b30d01bd98e06e3e P 3f75ed97c1aa4752b59f6a1551419390: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
Aug 12 05:23:35 dist-test-slave-42z9 krb5kdc[25677](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512215, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.100.1@KRBTEST.COM for kudu/127.31.100.62@KRBTEST.COM
I20260812 05:23:35.504948 32144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0waFvT/build/release/bin/kudu as pid 26779
I20260812 05:23:35.505236 32144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0waFvT/build/release/bin/kudu
/tmp/dist-test-task0waFvT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786512017613366-32144-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786512017613366-32144-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786512017613366-32144-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786512017613366-32144-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.100.2:39191
--local_ip_for_outbound_sockets=127.31.100.2
--tserver_master_addrs=127.31.100.62:37129
--webserver_port=40853
--webserver_interface=127.31.100.2
--keytab_file=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.2.keytab
--principal=kudu/127.31.100.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.100.20:45999
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786512017613366-32144-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786512017613366-32144-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786512017613366-32144-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 05:23:35.508832 27052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:23:35.496034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.62:37129 (local address 127.31.100.1:43107)
0812 05:23:35.497573 (+ 1539us) negotiation.cc:107] Waiting for socket to connect
0812 05:23:35.497621 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0812 05:23:35.498207 (+ 586us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:23:35.498708 (+ 501us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:23:35.498713 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:23:35.499075 (+ 362us) client_negotiation.cc:189] Negotiated authn=SASL
0812 05:23:35.499328 (+ 253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:23:35.499335 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:23:35.501003 (+ 1668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:23:35.501004 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:23:35.502267 (+ 1263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:23:35.502270 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:23:35.503134 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:23:35.503135 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:23:35.503271 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:23:35.503766 (+ 495us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 05:23:35.503775 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0812 05:23:35.505356 (+ 1581us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 05:23:35.507346 (+ 1990us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 05:23:35.507353 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 05:23:35.507355 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 05:23:35.507571 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 05:23:35.507802 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 05:23:35.507803 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 05:23:35.507804 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 05:23:35.507904 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 05:23:35.508233 (+ 329us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 05:23:35.508240 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 05:23:35.508392 (+ 152us) client_negotiation.cc:769] Sending connection context
0812 05:23:35.508489 (+ 97us) client_negotiation.cc:240] Negotiation successful
0812 05:23:35.508644 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1490,"thread_start_us":107,"threads_started":1}
I20260812 05:23:35.508984 27053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:23:35.496153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.100.1:43107 (local address 127.31.100.62:37129)
0812 05:23:35.497979 (+ 1826us) server_negotiation.cc:206] Beginning negotiation
0812 05:23:35.497987 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 05:23:35.498019 (+ 32us) server_negotiation.cc:407] Connection header received
0812 05:23:35.498426 (+ 407us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:23:35.498433 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:23:35.498511 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:23:35.498655 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:23:35.499481 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:23:35.500870 (+ 1389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:23:35.502357 (+ 1487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:23:35.503004 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:23:35.503117 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:23:35.505519 (+ 2402us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 05:23:35.505523 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 05:23:35.505523 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 05:23:35.505535 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0812 05:23:35.507273 (+ 1738us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:23:35.507656 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:23:35.507658 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:23:35.507659 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:23:35.507745 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:23:35.507968 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:23:35.507969 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:23:35.507969 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:23:35.508171 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 05:23:35.508239 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0812 05:23:35.508606 (+ 367us) server_negotiation.cc:299] Negotiation successful
0812 05:23:35.508755 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1776,"thread_start_us":88,"threads_started":1}
I20260812 05:23:35.509850 27050 heartbeater.cc:347] Connected to a master server at 127.31.100.62:37129
I20260812 05:23:35.509950 27050 heartbeater.cc:464] Registering TS with master...
I20260812 05:23:35.510118 27050 heartbeater.cc:511] Master 127.31.100.62:37129 requested a full tablet report, sending...
I20260812 05:23:35.510869 26705 ts_manager.cc:194] Registered new tserver with Master: 3f75ed97c1aa4752b59f6a1551419390 (127.31.100.1:37667)
I20260812 05:23:35.512099 26705 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.100.1@KRBTEST.COM'} at 127.31.100.1:43107
I20260812 05:23:35.513043 27050 heartbeater.cc:503] Master 127.31.100.62:37129 was elected leader, sending a full tablet report...
I20260812 05:23:35.520937 27062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:23:35.511542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.1:37667 (local address 127.0.0.1:40416)
0812 05:23:35.511955 (+ 413us) negotiation.cc:107] Waiting for socket to connect
0812 05:23:35.511990 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0812 05:23:35.512052 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:23:35.517727 (+ 5675us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:23:35.517730 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:23:35.517732 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 05:23:35.517793 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:23:35.517794 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:23:35.518981 (+ 1187us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:23:35.518982 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:23:35.519895 (+ 913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:23:35.519898 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:23:35.520726 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:23:35.520727 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:23:35.520791 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:23:35.520801 (+ 10us) client_negotiation.cc:769] Sending connection context
0812 05:23:35.520851 (+ 50us) client_negotiation.cc:240] Negotiation successful
0812 05:23:35.520888 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":386,"thread_start_us":85,"threads_started":1}
I20260812 05:23:35.520975 27063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:23:35.511778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40416 (local address 127.31.100.1:37667)
0812 05:23:35.517548 (+ 5770us) server_negotiation.cc:206] Beginning negotiation
0812 05:23:35.517552 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 05:23:35.517563 (+ 11us) server_negotiation.cc:407] Connection header received
0812 05:23:35.517594 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:23:35.517596 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:23:35.517611 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:23:35.517684 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 05:23:35.517856 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:23:35.518900 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:23:35.519964 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:23:35.520660 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:23:35.520727 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:23:35.520777 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0812 05:23:35.520875 (+ 98us) server_negotiation.cc:299] Negotiation successful
0812 05:23:35.520911 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5736,"thread_start_us":78,"threads_started":1}
I20260812 05:23:35.521288 26902 tablet_service.cc:1514] Processing DeleteTablet for tablet c3ca48be76d94bf9b30d01bd98e06e3e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 05:23:15 UTC) from {username='kudu', principal='kudu/127.31.100.62@KRBTEST.COM'} at 127.0.0.1:40416
I20260812 05:23:35.521772 27065 tablet_replica.cc:330] stopping tablet replica
I20260812 05:23:35.521860 27065 raft_consensus.cc:2229] T c3ca48be76d94bf9b30d01bd98e06e3e P 3f75ed97c1aa4752b59f6a1551419390 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 05:23:35.521925 27065 raft_consensus.cc:2258] T c3ca48be76d94bf9b30d01bd98e06e3e P 3f75ed97c1aa4752b59f6a1551419390 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 05:23:35.522188 27065 ts_tablet_manager.cc:1905] T c3ca48be76d94bf9b30d01bd98e06e3e P 3f75ed97c1aa4752b59f6a1551419390: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 05:23:35.523952 27065 ts_tablet_manager.cc:1918] T c3ca48be76d94bf9b30d01bd98e06e3e P 3f75ed97c1aa4752b59f6a1551419390: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 05:23:35.524009 27065 log.cc:1196] T c3ca48be76d94bf9b30d01bd98e06e3e P 3f75ed97c1aa4752b59f6a1551419390: Deleting WAL directory at /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786512017613366-32144-0/minicluster-data/ts-0/wal/wals/c3ca48be76d94bf9b30d01bd98e06e3e
I20260812 05:23:35.524204 27065 ts_tablet_manager.cc:1939] T c3ca48be76d94bf9b30d01bd98e06e3e P 3f75ed97c1aa4752b59f6a1551419390: Deleting consensus metadata
I20260812 05:23:35.524605 26680 catalog_manager.cc:4841] TS 3f75ed97c1aa4752b59f6a1551419390 (127.31.100.1:37667): tablet c3ca48be76d94bf9b30d01bd98e06e3e (table test-table [id=efa7108ddfc249af8fbe44e2d7d38c5e]) successfully deleted
W20260812 05:23:35.588454 27060 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 05:23:35.588656 27060 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 05:23:35.588685 27060 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 05:23:35.589802 27060 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 05:23:35.589843 27060 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 05:23:35.589875 27060 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.100.2
I20260812 05:23:35.591262 27060 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.100.20:45999
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786512017613366-32144-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786512017613366-32144-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.2.keytab
--principal=kudu/127.31.100.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.100.2:39191
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786512017613366-32144-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.100.2
--webserver_port=40853
--tserver_master_addrs=127.31.100.62:37129
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27060
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.100.2
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786512017613366-32144-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 04:20:34 UTC on ccycloud.quasar-phqdqp.root.comops.site
build id 4581871
I20260812 05:23:35.591500 27060 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 05:23:35.591692 27060 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 05:23:35.596374 27086 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 05:23:35.596374 27074 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 05:23:35.596519 27060 server_base.cc:1034] running on GCE node
W20260812 05:23:35.596402 27075 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 05:23:35.596786 27060 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 05:23:35.596983 27060 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 05:23:35.598119 27060 hybrid_clock.cc:648] HybridClock initialized: now 1786512215598102 us; error 26 us; skew 500 ppm
Aug 12 05:23:35 dist-test-slave-42z9 krb5kdc[25677](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512215, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.100.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 05:23:35.600427 27060 init.cc:377] Logged in from keytab as kudu/127.31.100.2@KRBTEST.COM (short username kudu)
I20260812 05:23:35.600782 27060 webserver.cc:460] Webserver started at http://127.31.100.2:40853/ using document root <none> and password file <none>
I20260812 05:23:35.600945 27060 fs_manager.cc:359] Metadata directory not provided
I20260812 05:23:35.601019 27060 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 05:23:35.602248 27060 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260812 05:23:35.602751 27098 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 05:23:35.602921 27060 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260812 05:23:35.603004 27060 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786512017613366-32144-0/minicluster-data/ts-1/data,/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786512017613366-32144-0/minicluster-data/ts-1/wal
uuid: "edf25888a7e648348650523730de9401"
format_stamp: "Formatted at 2026-08-12 05:23:14 on dist-test-slave-42z9"
server_key: "e08482d6fcbcc44b05a705170d41aa7f"
server_key_iv: "f08036fffbeee9c4ac9bcf06ad123959"
server_key_version: "encryptionkey@0"
I20260812 05:23:35.603282 27060 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786512017613366-32144-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786512017613366-32144-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786512017613366-32144-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 05:23:35.613473 27060 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 05:23:35.615022 27060 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 05:23:35.615124 27060 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 05:23:35.615286 27060 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 05:23:35.615522 27060 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 05:23:35.615552 27060 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 05:23:35.615571 27060 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 05:23:35.615586 27060 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 05:23:35.620850 27060 rpc_server.cc:307] RPC server started. Bound to: 127.31.100.2:39191
I20260812 05:23:35.620909 27296 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.100.2:39191 every 8 connection(s)
I20260812 05:23:35.621177 27060 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786512017613366-32144-0/minicluster-data/ts-1/data/info.pb
I20260812 05:23:35.621320 32144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0waFvT/build/release/bin/kudu as pid 27060
I20260812 05:23:35.621657 32144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0waFvT/build/release/bin/kudu
/tmp/dist-test-task0waFvT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786512017613366-32144-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786512017613366-32144-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786512017613366-32144-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786512017613366-32144-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.100.3:45871
--local_ip_for_outbound_sockets=127.31.100.3
--tserver_master_addrs=127.31.100.62:37129
--webserver_port=38899
--webserver_interface=127.31.100.3
--keytab_file=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.3.keytab
--principal=kudu/127.31.100.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.100.20:45999
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786512017613366-32144-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786512017613366-32144-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786512017613366-32144-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 12 05:23:35 dist-test-slave-42z9 krb5kdc[25677](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512215, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.100.2@KRBTEST.COM for kudu/127.31.100.62@KRBTEST.COM
I20260812 05:23:35.630740 27053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:23:35.621905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.100.2:33633 (local address 127.31.100.62:37129)
0812 05:23:35.621962 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0812 05:23:35.621965 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 05:23:35.622219 (+ 254us) server_negotiation.cc:407] Connection header received
0812 05:23:35.622765 (+ 546us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:23:35.622766 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:23:35.622780 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:23:35.622830 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:23:35.623861 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:23:35.624591 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:23:35.625876 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:23:35.626073 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:23:35.626122 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:23:35.628290 (+ 2168us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 05:23:35.628292 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 05:23:35.628292 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 05:23:35.628301 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0812 05:23:35.629552 (+ 1251us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:23:35.629895 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:23:35.629897 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:23:35.629897 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:23:35.629954 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:23:35.630163 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:23:35.630164 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:23:35.630164 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:23:35.630266 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 05:23:35.630321 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0812 05:23:35.630561 (+ 240us) server_negotiation.cc:299] Negotiation successful
0812 05:23:35.630651 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260812 05:23:35.630869 27300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:23:35.621912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.62:37129 (local address 127.31.100.2:33633)
0812 05:23:35.622092 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0812 05:23:35.622135 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0812 05:23:35.622676 (+ 541us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:23:35.622894 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:23:35.622897 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:23:35.623603 (+ 706us) client_negotiation.cc:189] Negotiated authn=SASL
0812 05:23:35.623776 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:23:35.623781 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:23:35.624674 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:23:35.624675 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:23:35.625804 (+ 1129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:23:35.625807 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:23:35.626154 (+ 347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:23:35.626155 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:23:35.626290 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:23:35.626733 (+ 443us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 05:23:35.626743 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0812 05:23:35.628203 (+ 1460us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 05:23:35.629618 (+ 1415us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 05:23:35.629626 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 05:23:35.629628 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 05:23:35.629832 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 05:23:35.630006 (+ 174us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 05:23:35.630007 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 05:23:35.630007 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 05:23:35.630108 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 05:23:35.630320 (+ 212us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 05:23:35.630327 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 05:23:35.630449 (+ 122us) client_negotiation.cc:769] Sending connection context
0812 05:23:35.630533 (+ 84us) client_negotiation.cc:240] Negotiation successful
0812 05:23:35.630673 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":87,"threads_started":1}
I20260812 05:23:35.631287 27298 heartbeater.cc:347] Connected to a master server at 127.31.100.62:37129
I20260812 05:23:35.631388 27298 heartbeater.cc:464] Registering TS with master...
I20260812 05:23:35.631572 27298 heartbeater.cc:511] Master 127.31.100.62:37129 requested a full tablet report, sending...
I20260812 05:23:35.631969 26705 ts_manager.cc:194] Registered new tserver with Master: edf25888a7e648348650523730de9401 (127.31.100.2:39191)
I20260812 05:23:35.632328 26705 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.100.2@KRBTEST.COM'} at 127.31.100.2:33633
W20260812 05:23:35.703449 27302 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 05:23:35.703644 27302 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 05:23:35.703665 27302 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 05:23:35.704726 27302 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 05:23:35.704759 27302 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 05:23:35.704783 27302 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.100.3
I20260812 05:23:35.706228 27302 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.100.20:45999
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786512017613366-32144-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786512017613366-32144-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.3.keytab
--principal=kudu/127.31.100.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.100.3:45871
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786512017613366-32144-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.100.3
--webserver_port=38899
--tserver_master_addrs=127.31.100.62:37129
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27302
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.100.3
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786512017613366-32144-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 04:20:34 UTC on ccycloud.quasar-phqdqp.root.comops.site
build id 4581871
I20260812 05:23:35.706408 27302 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 05:23:35.706580 27302 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 05:23:35.711279 27313 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 05:23:35.711289 27312 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 05:23:35.711452 27302 server_base.cc:1034] running on GCE node
W20260812 05:23:35.711289 27315 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 05:23:35.711719 27302 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 05:23:35.711932 27302 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 05:23:35.713065 27302 hybrid_clock.cc:648] HybridClock initialized: now 1786512215713054 us; error 24 us; skew 500 ppm
Aug 12 05:23:35 dist-test-slave-42z9 krb5kdc[25677](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512215, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.100.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 05:23:35.715456 27302 init.cc:377] Logged in from keytab as kudu/127.31.100.3@KRBTEST.COM (short username kudu)
I20260812 05:23:35.715871 27302 webserver.cc:460] Webserver started at http://127.31.100.3:38899/ using document root <none> and password file <none>
I20260812 05:23:35.716074 27302 fs_manager.cc:359] Metadata directory not provided
I20260812 05:23:35.716135 27302 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 05:23:35.717372 27302 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260812 05:23:35.717960 27327 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 05:23:35.718102 27302 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260812 05:23:35.718184 27302 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786512017613366-32144-0/minicluster-data/ts-2/data,/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786512017613366-32144-0/minicluster-data/ts-2/wal
uuid: "df8db312fb6c4c57aa51f490c25fa09f"
format_stamp: "Formatted at 2026-08-12 05:23:14 on dist-test-slave-42z9"
server_key: "9ce08b8ccb671a16de628348a0a75d2a"
server_key_iv: "186a8874839af3036fc47945c19039fb"
server_key_version: "encryptionkey@0"
I20260812 05:23:35.718464 27302 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786512017613366-32144-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786512017613366-32144-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786512017613366-32144-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 05:23:35.738646 27302 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 05:23:35.740370 27302 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 05:23:35.740476 27302 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 05:23:35.740640 27302 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 05:23:35.740882 27302 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 05:23:35.740918 27302 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 05:23:35.740938 27302 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 05:23:35.740953 27302 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 05:23:35.746320 27302 rpc_server.cc:307] RPC server started. Bound to: 127.31.100.3:45871
I20260812 05:23:35.746380 27518 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.100.3:45871 every 8 connection(s)
I20260812 05:23:35.746707 27302 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786512017613366-32144-0/minicluster-data/ts-2/data/info.pb
I20260812 05:23:35.747820 32144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0waFvT/build/release/bin/kudu as pid 27302
Aug 12 05:23:35 dist-test-slave-42z9 krb5kdc[25677](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512215, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.100.3@KRBTEST.COM for kudu/127.31.100.62@KRBTEST.COM
I20260812 05:23:35.754395 27529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:23:35.748253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51150 (local address 127.31.100.62:37129)
0812 05:23:35.748382 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0812 05:23:35.748384 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 05:23:35.748538 (+ 154us) server_negotiation.cc:407] Connection header received
0812 05:23:35.748599 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:23:35.748600 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:23:35.748611 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:23:35.748667 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:23:35.749244 (+ 577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:23:35.749815 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:23:35.750553 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:23:35.750738 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:23:35.750790 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:23:35.751948 (+ 1158us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 05:23:35.751950 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 05:23:35.751950 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 05:23:35.751956 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0812 05:23:35.753322 (+ 1366us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:23:35.753665 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:23:35.753666 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:23:35.753667 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:23:35.753719 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:23:35.753883 (+ 164us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:23:35.753884 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:23:35.753884 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:23:35.753983 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 05:23:35.754148 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0812 05:23:35.754184 (+ 36us) server_negotiation.cc:299] Negotiation successful
0812 05:23:35.754304 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":20,"server-negotiator.queue_time_us":111,"thread_start_us":79,"threads_started":1}
I20260812 05:23:35.755637 27053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:23:35.747399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.100.3:60743 (local address 127.31.100.62:37129)
0812 05:23:35.747478 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0812 05:23:35.747479 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0812 05:23:35.747648 (+ 169us) server_negotiation.cc:407] Connection header received
0812 05:23:35.748298 (+ 650us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:23:35.748300 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:23:35.748312 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:23:35.748376 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:23:35.749285 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:23:35.749816 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:23:35.750740 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:23:35.750935 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:23:35.750988 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:23:35.753036 (+ 2048us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 05:23:35.753037 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 05:23:35.753038 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 05:23:35.753044 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0812 05:23:35.754356 (+ 1312us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:23:35.754795 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:23:35.754796 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:23:35.754796 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:23:35.754858 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:23:35.755127 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:23:35.755127 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:23:35.755128 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:23:35.755251 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 05:23:35.755308 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0812 05:23:35.755520 (+ 212us) server_negotiation.cc:299] Negotiation successful
0812 05:23:35.755582 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":50}
I20260812 05:23:35.755802 27522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:23:35.747343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.62:37129 (local address 127.31.100.3:60743)
0812 05:23:35.747504 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0812 05:23:35.747546 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0812 05:23:35.748167 (+ 621us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:23:35.748460 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:23:35.748465 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:23:35.749007 (+ 542us) client_negotiation.cc:189] Negotiated authn=SASL
0812 05:23:35.749197 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:23:35.749201 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:23:35.749892 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:23:35.749893 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:23:35.750664 (+ 771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:23:35.750666 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:23:35.751000 (+ 334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:23:35.751001 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:23:35.751121 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:23:35.751447 (+ 326us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 05:23:35.751454 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0812 05:23:35.752939 (+ 1485us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 05:23:35.754433 (+ 1494us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 05:23:35.754440 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 05:23:35.754443 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 05:23:35.754718 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 05:23:35.754917 (+ 199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 05:23:35.754917 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 05:23:35.754918 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 05:23:35.755064 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 05:23:35.755311 (+ 247us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 05:23:35.755315 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 05:23:35.755420 (+ 105us) client_negotiation.cc:769] Sending connection context
0812 05:23:35.755507 (+ 87us) client_negotiation.cc:240] Negotiation successful
0812 05:23:35.755619 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":94,"threads_started":1}
I20260812 05:23:35.756202 27520 heartbeater.cc:347] Connected to a master server at 127.31.100.62:37129
I20260812 05:23:35.756284 27520 heartbeater.cc:464] Registering TS with master...
I20260812 05:23:35.756417 27520 heartbeater.cc:511] Master 127.31.100.62:37129 requested a full tablet report, sending...
I20260812 05:23:35.756719 26705 ts_manager.cc:194] Registered new tserver with Master: df8db312fb6c4c57aa51f490c25fa09f (127.31.100.3:45871)
I20260812 05:23:35.757030 26705 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.100.3@KRBTEST.COM'} at 127.31.100.3:60743
I20260812 05:23:35.765909 32144 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 05:23:35.768086 27053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:23:35.766186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51166 (local address 127.31.100.62:37129)
0812 05:23:35.766230 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0812 05:23:35.766232 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 05:23:35.766322 (+ 90us) server_negotiation.cc:407] Connection header received
0812 05:23:35.766366 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:23:35.766367 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:23:35.766395 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:23:35.766445 (+ 50us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 05:23:35.766521 (+ 76us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:23:35.766940 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:23:35.767588 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:23:35.767748 (+ 160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:23:35.767787 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:23:35.767885 (+ 98us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 05:23:35.767922 (+ 37us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0812 05:23:35.768035 (+ 113us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:51166: token expired
Metrics: {"server-negotiator.queue_time_us":24}
W20260812 05:23:35.768141 27053 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:51166: token expired
W20260812 05:23:35.768286 26584 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.31.100.62:37129)
I20260812 05:23:35.773187 27053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:23:35.768525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51172 (local address 127.31.100.62:37129)
0812 05:23:35.768562 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0812 05:23:35.768564 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 05:23:35.768587 (+ 23us) server_negotiation.cc:407] Connection header received
0812 05:23:35.768637 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:23:35.768638 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:23:35.768644 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:23:35.768685 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:23:35.769096 (+ 411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:23:35.769571 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:23:35.770191 (+ 620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:23:35.770370 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:23:35.770410 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:23:35.771339 (+ 929us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 05:23:35.771340 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 05:23:35.771341 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 05:23:35.771346 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0812 05:23:35.772463 (+ 1117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:23:35.772700 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:23:35.772700 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:23:35.772701 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:23:35.772740 (+ 39us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:23:35.772841 (+ 101us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:23:35.772842 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:23:35.772842 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:23:35.772942 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 05:23:35.773012 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0812 05:23:35.773082 (+ 70us) server_negotiation.cc:299] Negotiation successful
0812 05:23:35.773134 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260812 05:23:35.774279 26705 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51172:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 05:23:35.775751 26705 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 05:23:35.777720 26902 tablet_service.cc:1467] Processing CreateTablet for tablet 4caedac3e22b4e589962ff7140e69aaa (DEFAULT_TABLE table=test-table [id=51cf1888c0b043a58dad08df4823144f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 05:23:35.777925 26902 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4caedac3e22b4e589962ff7140e69aaa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 05:23:35.779223 26853 tablet_bootstrap.cc:492] T 4caedac3e22b4e589962ff7140e69aaa P 3f75ed97c1aa4752b59f6a1551419390: Bootstrap starting.
I20260812 05:23:35.779570 26853 tablet_bootstrap.cc:654] T 4caedac3e22b4e589962ff7140e69aaa P 3f75ed97c1aa4752b59f6a1551419390: Neither blocks nor log segments found. Creating new log.
I20260812 05:23:35.780143 26853 tablet_bootstrap.cc:492] T 4caedac3e22b4e589962ff7140e69aaa P 3f75ed97c1aa4752b59f6a1551419390: No bootstrap required, opened a new log
I20260812 05:23:35.780190 26853 ts_tablet_manager.cc:1397] T 4caedac3e22b4e589962ff7140e69aaa P 3f75ed97c1aa4752b59f6a1551419390: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260812 05:23:35.780287 26853 raft_consensus.cc:348] T 4caedac3e22b4e589962ff7140e69aaa P 3f75ed97c1aa4752b59f6a1551419390 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df8db312fb6c4c57aa51f490c25fa09f" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 45871 } } peers { permanent_uuid: "edf25888a7e648348650523730de9401" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 39191 } } peers { permanent_uuid: "3f75ed97c1aa4752b59f6a1551419390" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 37667 } }
I20260812 05:23:35.780342 26853 raft_consensus.cc:374] T 4caedac3e22b4e589962ff7140e69aaa P 3f75ed97c1aa4752b59f6a1551419390 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 05:23:35.780381 26853 raft_consensus.cc:729] T 4caedac3e22b4e589962ff7140e69aaa P 3f75ed97c1aa4752b59f6a1551419390 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f75ed97c1aa4752b59f6a1551419390, State: Initialized, Role: FOLLOWER
I20260812 05:23:35.780426 26853 consensus_queue.cc:260] T 4caedac3e22b4e589962ff7140e69aaa P 3f75ed97c1aa4752b59f6a1551419390 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df8db312fb6c4c57aa51f490c25fa09f" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 45871 } } peers { permanent_uuid: "edf25888a7e648348650523730de9401" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 39191 } } peers { permanent_uuid: "3f75ed97c1aa4752b59f6a1551419390" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 37667 } }
I20260812 05:23:35.780509 26853 ts_tablet_manager.cc:1428] T 4caedac3e22b4e589962ff7140e69aaa P 3f75ed97c1aa4752b59f6a1551419390: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260812 05:23:35.781221 27532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:23:35.777610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.2:39191 (local address 127.0.0.1:39552)
0812 05:23:35.777752 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0812 05:23:35.777760 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0812 05:23:35.777802 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:23:35.778157 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:23:35.778158 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:23:35.778160 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 05:23:35.778202 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:23:35.778203 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:23:35.779176 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:23:35.779177 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:23:35.780106 (+ 929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:23:35.780109 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:23:35.780990 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:23:35.780991 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:23:35.781056 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:23:35.781059 (+ 3us) client_negotiation.cc:769] Sending connection context
0812 05:23:35.781112 (+ 53us) client_negotiation.cc:240] Negotiation successful
0812 05:23:35.781150 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":83,"threads_started":1}
I20260812 05:23:35.781234 27534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:23:35.777849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39552 (local address 127.31.100.2:39191)
0812 05:23:35.778024 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0812 05:23:35.778028 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 05:23:35.778036 (+ 8us) server_negotiation.cc:407] Connection header received
0812 05:23:35.778066 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:23:35.778067 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:23:35.778079 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:23:35.778148 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 05:23:35.778255 (+ 107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:23:35.779106 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:23:35.780179 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:23:35.780835 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:23:35.780901 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:23:35.780956 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0812 05:23:35.781131 (+ 175us) server_negotiation.cc:299] Negotiation successful
0812 05:23:35.781175 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":102,"threads_started":1}
I20260812 05:23:35.782233 27181 tablet_service.cc:1467] Processing CreateTablet for tablet 4caedac3e22b4e589962ff7140e69aaa (DEFAULT_TABLE table=test-table [id=51cf1888c0b043a58dad08df4823144f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 05:23:35.782509 27062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:23:35.777516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.3:45871 (local address 127.0.0.1:46744)
0812 05:23:35.777550 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0812 05:23:35.777559 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0812 05:23:35.777589 (+ 30us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:23:35.779573 (+ 1984us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:23:35.779574 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:23:35.779576 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 05:23:35.779605 (+ 29us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:23:35.779607 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:23:35.780658 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:23:35.780659 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:23:35.781606 (+ 947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:23:35.781609 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:23:35.782290 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:23:35.782291 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:23:35.782360 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:23:35.782363 (+ 3us) client_negotiation.cc:769] Sending connection context
0812 05:23:35.782414 (+ 51us) client_negotiation.cc:240] Negotiation successful
0812 05:23:35.782466 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17}
I20260812 05:23:35.782555 27181 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4caedac3e22b4e589962ff7140e69aaa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 05:23:35.782574 27533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:23:35.777605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46744 (local address 127.31.100.3:45871)
0812 05:23:35.779408 (+ 1803us) server_negotiation.cc:206] Beginning negotiation
0812 05:23:35.779413 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 05:23:35.779424 (+ 11us) server_negotiation.cc:407] Connection header received
0812 05:23:35.779466 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:23:35.779468 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:23:35.779483 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:23:35.779567 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 05:23:35.779663 (+ 96us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:23:35.780578 (+ 915us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:23:35.781663 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:23:35.782215 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:23:35.782280 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:23:35.782332 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0812 05:23:35.782454 (+ 122us) server_negotiation.cc:299] Negotiation successful
0812 05:23:35.782505 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1764,"thread_start_us":98,"threads_started":1}
I20260812 05:23:35.783111 27417 tablet_service.cc:1467] Processing CreateTablet for tablet 4caedac3e22b4e589962ff7140e69aaa (DEFAULT_TABLE table=test-table [id=51cf1888c0b043a58dad08df4823144f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 05:23:35.783438 27417 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4caedac3e22b4e589962ff7140e69aaa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 05:23:35.785239 27536 tablet_bootstrap.cc:492] T 4caedac3e22b4e589962ff7140e69aaa P edf25888a7e648348650523730de9401: Bootstrap starting.
I20260812 05:23:35.785923 27536 tablet_bootstrap.cc:654] T 4caedac3e22b4e589962ff7140e69aaa P edf25888a7e648348650523730de9401: Neither blocks nor log segments found. Creating new log.
I20260812 05:23:35.786140 27538 tablet_bootstrap.cc:492] T 4caedac3e22b4e589962ff7140e69aaa P df8db312fb6c4c57aa51f490c25fa09f: Bootstrap starting.
I20260812 05:23:35.786509 27536 log.cc:824] T 4caedac3e22b4e589962ff7140e69aaa P edf25888a7e648348650523730de9401: Log is configured to *not* fsync() on all Append() calls
I20260812 05:23:35.786701 27538 tablet_bootstrap.cc:654] T 4caedac3e22b4e589962ff7140e69aaa P df8db312fb6c4c57aa51f490c25fa09f: Neither blocks nor log segments found. Creating new log.
I20260812 05:23:35.787146 27536 tablet_bootstrap.cc:492] T 4caedac3e22b4e589962ff7140e69aaa P edf25888a7e648348650523730de9401: No bootstrap required, opened a new log
I20260812 05:23:35.787158 27538 log.cc:824] T 4caedac3e22b4e589962ff7140e69aaa P df8db312fb6c4c57aa51f490c25fa09f: Log is configured to *not* fsync() on all Append() calls
I20260812 05:23:35.787201 27536 ts_tablet_manager.cc:1397] T 4caedac3e22b4e589962ff7140e69aaa P edf25888a7e648348650523730de9401: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260812 05:23:35.787799 27538 tablet_bootstrap.cc:492] T 4caedac3e22b4e589962ff7140e69aaa P df8db312fb6c4c57aa51f490c25fa09f: No bootstrap required, opened a new log
I20260812 05:23:35.787858 27538 ts_tablet_manager.cc:1397] T 4caedac3e22b4e589962ff7140e69aaa P df8db312fb6c4c57aa51f490c25fa09f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260812 05:23:35.788642 27536 raft_consensus.cc:348] T 4caedac3e22b4e589962ff7140e69aaa P edf25888a7e648348650523730de9401 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df8db312fb6c4c57aa51f490c25fa09f" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 45871 } } peers { permanent_uuid: "edf25888a7e648348650523730de9401" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 39191 } } peers { permanent_uuid: "3f75ed97c1aa4752b59f6a1551419390" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 37667 } }
I20260812 05:23:35.788771 27536 raft_consensus.cc:374] T 4caedac3e22b4e589962ff7140e69aaa P edf25888a7e648348650523730de9401 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 05:23:35.788791 27536 raft_consensus.cc:729] T 4caedac3e22b4e589962ff7140e69aaa P edf25888a7e648348650523730de9401 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: edf25888a7e648348650523730de9401, State: Initialized, Role: FOLLOWER
I20260812 05:23:35.788884 27536 consensus_queue.cc:260] T 4caedac3e22b4e589962ff7140e69aaa P edf25888a7e648348650523730de9401 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df8db312fb6c4c57aa51f490c25fa09f" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 45871 } } peers { permanent_uuid: "edf25888a7e648348650523730de9401" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 39191 } } peers { permanent_uuid: "3f75ed97c1aa4752b59f6a1551419390" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 37667 } }
I20260812 05:23:35.789142 27536 ts_tablet_manager.cc:1428] T 4caedac3e22b4e589962ff7140e69aaa P edf25888a7e648348650523730de9401: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260812 05:23:35.789168 27298 heartbeater.cc:503] Master 127.31.100.62:37129 was elected leader, sending a full tablet report...
I20260812 05:23:35.789270 27538 raft_consensus.cc:348] T 4caedac3e22b4e589962ff7140e69aaa P df8db312fb6c4c57aa51f490c25fa09f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df8db312fb6c4c57aa51f490c25fa09f" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 45871 } } peers { permanent_uuid: "edf25888a7e648348650523730de9401" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 39191 } } peers { permanent_uuid: "3f75ed97c1aa4752b59f6a1551419390" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 37667 } }
I20260812 05:23:35.789425 27538 raft_consensus.cc:374] T 4caedac3e22b4e589962ff7140e69aaa P df8db312fb6c4c57aa51f490c25fa09f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 05:23:35.789527 27538 raft_consensus.cc:729] T 4caedac3e22b4e589962ff7140e69aaa P df8db312fb6c4c57aa51f490c25fa09f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df8db312fb6c4c57aa51f490c25fa09f, State: Initialized, Role: FOLLOWER
I20260812 05:23:35.789649 27538 consensus_queue.cc:260] T 4caedac3e22b4e589962ff7140e69aaa P df8db312fb6c4c57aa51f490c25fa09f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df8db312fb6c4c57aa51f490c25fa09f" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 45871 } } peers { permanent_uuid: "edf25888a7e648348650523730de9401" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 39191 } } peers { permanent_uuid: "3f75ed97c1aa4752b59f6a1551419390" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 37667 } }
I20260812 05:23:35.789923 27538 ts_tablet_manager.cc:1428] T 4caedac3e22b4e589962ff7140e69aaa P df8db312fb6c4c57aa51f490c25fa09f: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260812 05:23:35.789933 27520 heartbeater.cc:503] Master 127.31.100.62:37129 was elected leader, sending a full tablet report...
I20260812 05:23:35.831717 27057 raft_consensus.cc:482] T 4caedac3e22b4e589962ff7140e69aaa P 3f75ed97c1aa4752b59f6a1551419390 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 05:23:35.831840 27057 raft_consensus.cc:504] T 4caedac3e22b4e589962ff7140e69aaa P 3f75ed97c1aa4752b59f6a1551419390 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df8db312fb6c4c57aa51f490c25fa09f" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 45871 } } peers { permanent_uuid: "edf25888a7e648348650523730de9401" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 39191 } } peers { permanent_uuid: "3f75ed97c1aa4752b59f6a1551419390" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 37667 } }
I20260812 05:23:35.832126 27057 leader_election.cc:288] T 4caedac3e22b4e589962ff7140e69aaa P 3f75ed97c1aa4752b59f6a1551419390 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df8db312fb6c4c57aa51f490c25fa09f (127.31.100.3:45871), edf25888a7e648348650523730de9401 (127.31.100.2:39191)
I20260812 05:23:35.834997 27533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:23:35.832377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.100.1:35167 (local address 127.31.100.3:45871)
0812 05:23:35.832441 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0812 05:23:35.832443 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 05:23:35.832449 (+ 6us) server_negotiation.cc:407] Connection header received
0812 05:23:35.832473 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:23:35.832474 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:23:35.832491 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:23:35.832541 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 05:23:35.832910 (+ 369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:23:35.833352 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:23:35.834312 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:23:35.834679 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:23:35.834742 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:23:35.834750 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0812 05:23:35.834890 (+ 140us) server_negotiation.cc:299] Negotiation successful
0812 05:23:35.834932 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260812 05:23:35.835012 27052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:23:35.832279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.3:45871 (local address 127.31.100.1:35167)
0812 05:23:35.832333 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0812 05:23:35.832346 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 05:23:35.832396 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:23:35.832550 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:23:35.832551 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:23:35.832552 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 05:23:35.832842 (+ 290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:23:35.832844 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:23:35.833421 (+ 577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:23:35.833422 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:23:35.834230 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:23:35.834232 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:23:35.834747 (+ 515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:23:35.834748 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:23:35.834830 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:23:35.834833 (+ 3us) client_negotiation.cc:769] Sending connection context
0812 05:23:35.834881 (+ 48us) client_negotiation.cc:240] Negotiation successful
0812 05:23:35.834928 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27,"mutex_wait_us":39}
I20260812 05:23:35.835019 27534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:23:35.832446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.100.1:38835 (local address 127.31.100.2:39191)
0812 05:23:35.832494 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0812 05:23:35.832496 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 05:23:35.832604 (+ 108us) server_negotiation.cc:407] Connection header received
0812 05:23:35.832653 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:23:35.832654 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:23:35.832663 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:23:35.832711 (+ 48us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 05:23:35.832844 (+ 133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:23:35.833310 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:23:35.834317 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:23:35.834687 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:23:35.834749 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:23:35.834757 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0812 05:23:35.834928 (+ 171us) server_negotiation.cc:299] Negotiation successful
0812 05:23:35.834967 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260812 05:23:35.835258 27551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:23:35.832375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.2:39191 (local address 127.31.100.1:38835)
0812 05:23:35.832539 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0812 05:23:35.832547 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0812 05:23:35.832602 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:23:35.832722 (+ 120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:23:35.832723 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:23:35.832724 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 05:23:35.832791 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:23:35.832792 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:23:35.833385 (+ 593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:23:35.833386 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:23:35.834231 (+ 845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:23:35.834233 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:23:35.834773 (+ 540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:23:35.834774 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:23:35.834841 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:23:35.834845 (+ 4us) client_negotiation.cc:769] Sending connection context
0812 05:23:35.835130 (+ 285us) client_negotiation.cc:240] Negotiation successful
0812 05:23:35.835174 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":114,"threads_started":1}
I20260812 05:23:35.835428 27451 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4caedac3e22b4e589962ff7140e69aaa" candidate_uuid: "3f75ed97c1aa4752b59f6a1551419390" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df8db312fb6c4c57aa51f490c25fa09f" is_pre_election: true
I20260812 05:23:35.835561 27217 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4caedac3e22b4e589962ff7140e69aaa" candidate_uuid: "3f75ed97c1aa4752b59f6a1551419390" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "edf25888a7e648348650523730de9401" is_pre_election: true
I20260812 05:23:35.835613 27451 raft_consensus.cc:2454] T 4caedac3e22b4e589962ff7140e69aaa P df8db312fb6c4c57aa51f490c25fa09f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f75ed97c1aa4752b59f6a1551419390 in term 0.
I20260812 05:23:35.835691 27217 raft_consensus.cc:2454] T 4caedac3e22b4e589962ff7140e69aaa P edf25888a7e648348650523730de9401 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f75ed97c1aa4752b59f6a1551419390 in term 0.
I20260812 05:23:35.835825 26845 leader_election.cc:302] T 4caedac3e22b4e589962ff7140e69aaa P 3f75ed97c1aa4752b59f6a1551419390 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f75ed97c1aa4752b59f6a1551419390, df8db312fb6c4c57aa51f490c25fa09f; no voters:
I20260812 05:23:35.835949 27057 raft_consensus.cc:2781] T 4caedac3e22b4e589962ff7140e69aaa P 3f75ed97c1aa4752b59f6a1551419390 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 05:23:35.836017 27057 raft_consensus.cc:482] T 4caedac3e22b4e589962ff7140e69aaa P 3f75ed97c1aa4752b59f6a1551419390 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 05:23:35.836041 27057 raft_consensus.cc:3037] T 4caedac3e22b4e589962ff7140e69aaa P 3f75ed97c1aa4752b59f6a1551419390 [term 0 FOLLOWER]: Advancing to term 1
I20260812 05:23:35.836702 27057 raft_consensus.cc:504] T 4caedac3e22b4e589962ff7140e69aaa P 3f75ed97c1aa4752b59f6a1551419390 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df8db312fb6c4c57aa51f490c25fa09f" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 45871 } } peers { permanent_uuid: "edf25888a7e648348650523730de9401" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 39191 } } peers { permanent_uuid: "3f75ed97c1aa4752b59f6a1551419390" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 37667 } }
I20260812 05:23:35.836822 27057 leader_election.cc:288] T 4caedac3e22b4e589962ff7140e69aaa P 3f75ed97c1aa4752b59f6a1551419390 [CANDIDATE]: Term 1 election: Requested vote from peers df8db312fb6c4c57aa51f490c25fa09f (127.31.100.3:45871), edf25888a7e648348650523730de9401 (127.31.100.2:39191)
I20260812 05:23:35.836967 27451 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4caedac3e22b4e589962ff7140e69aaa" candidate_uuid: "3f75ed97c1aa4752b59f6a1551419390" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df8db312fb6c4c57aa51f490c25fa09f"
I20260812 05:23:35.837014 27217 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4caedac3e22b4e589962ff7140e69aaa" candidate_uuid: "3f75ed97c1aa4752b59f6a1551419390" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "edf25888a7e648348650523730de9401"
I20260812 05:23:35.837049 27451 raft_consensus.cc:3037] T 4caedac3e22b4e589962ff7140e69aaa P df8db312fb6c4c57aa51f490c25fa09f [term 0 FOLLOWER]: Advancing to term 1
I20260812 05:23:35.837077 27217 raft_consensus.cc:3037] T 4caedac3e22b4e589962ff7140e69aaa P edf25888a7e648348650523730de9401 [term 0 FOLLOWER]: Advancing to term 1
I20260812 05:23:35.837935 27217 raft_consensus.cc:2454] T 4caedac3e22b4e589962ff7140e69aaa P edf25888a7e648348650523730de9401 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f75ed97c1aa4752b59f6a1551419390 in term 1.
I20260812 05:23:35.837935 27451 raft_consensus.cc:2454] T 4caedac3e22b4e589962ff7140e69aaa P df8db312fb6c4c57aa51f490c25fa09f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f75ed97c1aa4752b59f6a1551419390 in term 1.
I20260812 05:23:35.838112 26845 leader_election.cc:302] T 4caedac3e22b4e589962ff7140e69aaa P 3f75ed97c1aa4752b59f6a1551419390 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f75ed97c1aa4752b59f6a1551419390, edf25888a7e648348650523730de9401; no voters:
I20260812 05:23:35.838199 27057 raft_consensus.cc:2781] T 4caedac3e22b4e589962ff7140e69aaa P 3f75ed97c1aa4752b59f6a1551419390 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 05:23:35.838340 27057 raft_consensus.cc:686] T 4caedac3e22b4e589962ff7140e69aaa P 3f75ed97c1aa4752b59f6a1551419390 [term 1 LEADER]: Becoming Leader. State: Replica: 3f75ed97c1aa4752b59f6a1551419390, State: Running, Role: LEADER
I20260812 05:23:35.838403 27057 consensus_queue.cc:237] T 4caedac3e22b4e589962ff7140e69aaa P 3f75ed97c1aa4752b59f6a1551419390 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df8db312fb6c4c57aa51f490c25fa09f" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 45871 } } peers { permanent_uuid: "edf25888a7e648348650523730de9401" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 39191 } } peers { permanent_uuid: "3f75ed97c1aa4752b59f6a1551419390" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 37667 } }
I20260812 05:23:35.839130 26705 catalog_manager.cc:5495] T 4caedac3e22b4e589962ff7140e69aaa P 3f75ed97c1aa4752b59f6a1551419390 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3f75ed97c1aa4752b59f6a1551419390 (127.31.100.1). New cstate: current_term: 1 leader_uuid: "3f75ed97c1aa4752b59f6a1551419390" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df8db312fb6c4c57aa51f490c25fa09f" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 45871 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "edf25888a7e648348650523730de9401" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 39191 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3f75ed97c1aa4752b59f6a1551419390" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 37667 } health_report { overall_health: HEALTHY } } }
I20260812 05:23:35.853497 27063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:23:35.851589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40426 (local address 127.31.100.1:37667)
0812 05:23:35.851624 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0812 05:23:35.851626 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 05:23:35.851655 (+ 29us) server_negotiation.cc:407] Connection header received
0812 05:23:35.851710 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:23:35.851711 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:23:35.851725 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:23:35.851780 (+ 55us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 05:23:35.851868 (+ 88us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:23:35.852300 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:23:35.852931 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:23:35.853103 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:23:35.853142 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:23:35.853239 (+ 97us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 05:23:35.853324 (+ 85us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 05:23:35.853369 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0812 05:23:35.853400 (+ 31us) server_negotiation.cc:299] Negotiation successful
0812 05:23:35.853416 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260812 05:23:35.854832 27451 raft_consensus.cc:1261] T 4caedac3e22b4e589962ff7140e69aaa P df8db312fb6c4c57aa51f490c25fa09f [term 1 FOLLOWER]: Refusing update from remote peer 3f75ed97c1aa4752b59f6a1551419390: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 05:23:35.854831 27217 raft_consensus.cc:1261] T 4caedac3e22b4e589962ff7140e69aaa P edf25888a7e648348650523730de9401 [term 1 FOLLOWER]: Refusing update from remote peer 3f75ed97c1aa4752b59f6a1551419390: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 05:23:35.855149 27057 consensus_queue.cc:1035] T 4caedac3e22b4e589962ff7140e69aaa P 3f75ed97c1aa4752b59f6a1551419390 [LEADER]: Connected to new peer: Peer: permanent_uuid: "edf25888a7e648348650523730de9401" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 39191 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 05:23:35.855288 27554 consensus_queue.cc:1035] T 4caedac3e22b4e589962ff7140e69aaa P 3f75ed97c1aa4752b59f6a1551419390 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df8db312fb6c4c57aa51f490c25fa09f" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 45871 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 05:23:35.857512 27559 mvcc.cc:205] Tried to move back new op lower bound from 7317554036139311104 to 7317554036074635264. Current Snapshot: MvccSnapshot[applied={T|T < 7317554036139311104}]
I20260812 05:23:35.859762 26703 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51172:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 05:23:35.859844 26703 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51172:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 05:23:35.860509 26703 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a2ede0bc8af24780a89103af190035b8: Sending DeleteTablet for 3 replicas of tablet 4caedac3e22b4e589962ff7140e69aaa
I20260812 05:23:35.860814 27181 tablet_service.cc:1514] Processing DeleteTablet for tablet 4caedac3e22b4e589962ff7140e69aaa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 05:23:35 UTC) from {username='kudu', principal='kudu/127.31.100.62@KRBTEST.COM'} at 127.0.0.1:39552
I20260812 05:23:35.860848 32144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0waFvT/build/release/bin/kudu with pid 26779
I20260812 05:23:35.860822 27417 tablet_service.cc:1514] Processing DeleteTablet for tablet 4caedac3e22b4e589962ff7140e69aaa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 05:23:35 UTC) from {username='kudu', principal='kudu/127.31.100.62@KRBTEST.COM'} at 127.0.0.1:46744
I20260812 05:23:35.860875 26902 tablet_service.cc:1514] Processing DeleteTablet for tablet 4caedac3e22b4e589962ff7140e69aaa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 05:23:35 UTC) from {username='kudu', principal='kudu/127.31.100.62@KRBTEST.COM'} at 127.0.0.1:40416
I20260812 05:23:35.869313 27582 tablet_replica.cc:330] T 4caedac3e22b4e589962ff7140e69aaa P df8db312fb6c4c57aa51f490c25fa09f: stopping tablet replica
I20260812 05:23:35.869415 27582 raft_consensus.cc:2229] T 4caedac3e22b4e589962ff7140e69aaa P df8db312fb6c4c57aa51f490c25fa09f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 05:23:35.869518 27582 raft_consensus.cc:2258] T 4caedac3e22b4e589962ff7140e69aaa P df8db312fb6c4c57aa51f490c25fa09f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 05:23:35.870445 27582 ts_tablet_manager.cc:1905] T 4caedac3e22b4e589962ff7140e69aaa P df8db312fb6c4c57aa51f490c25fa09f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 05:23:35.871961 27582 ts_tablet_manager.cc:1918] T 4caedac3e22b4e589962ff7140e69aaa P df8db312fb6c4c57aa51f490c25fa09f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 05:23:35.872080 27582 log.cc:1196] T 4caedac3e22b4e589962ff7140e69aaa P df8db312fb6c4c57aa51f490c25fa09f: Deleting WAL directory at /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786512017613366-32144-0/minicluster-data/ts-2/wal/wals/4caedac3e22b4e589962ff7140e69aaa
I20260812 05:23:35.872361 27582 ts_tablet_manager.cc:1939] T 4caedac3e22b4e589962ff7140e69aaa P df8db312fb6c4c57aa51f490c25fa09f: Deleting consensus metadata
I20260812 05:23:35.872776 26678 catalog_manager.cc:4841] TS df8db312fb6c4c57aa51f490c25fa09f (127.31.100.3:45871): tablet 4caedac3e22b4e589962ff7140e69aaa (table test-table [id=51cf1888c0b043a58dad08df4823144f]) successfully deleted
I20260812 05:23:35.873009 27581 tablet_replica.cc:330] T 4caedac3e22b4e589962ff7140e69aaa P edf25888a7e648348650523730de9401: stopping tablet replica
I20260812 05:23:35.873102 27581 raft_consensus.cc:2229] T 4caedac3e22b4e589962ff7140e69aaa P edf25888a7e648348650523730de9401 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 05:23:35.873176 27581 raft_consensus.cc:2258] T 4caedac3e22b4e589962ff7140e69aaa P edf25888a7e648348650523730de9401 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 05:23:35.873994 27581 ts_tablet_manager.cc:1905] T 4caedac3e22b4e589962ff7140e69aaa P edf25888a7e648348650523730de9401: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 05:23:35.875104 27581 ts_tablet_manager.cc:1918] T 4caedac3e22b4e589962ff7140e69aaa P edf25888a7e648348650523730de9401: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 05:23:35.875167 27581 log.cc:1196] T 4caedac3e22b4e589962ff7140e69aaa P edf25888a7e648348650523730de9401: Deleting WAL directory at /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786512017613366-32144-0/minicluster-data/ts-1/wal/wals/4caedac3e22b4e589962ff7140e69aaa
I20260812 05:23:35.875377 27581 ts_tablet_manager.cc:1939] T 4caedac3e22b4e589962ff7140e69aaa P edf25888a7e648348650523730de9401: Deleting consensus metadata
I20260812 05:23:35.876237 26678 catalog_manager.cc:4841] TS edf25888a7e648348650523730de9401 (127.31.100.2:39191): tablet 4caedac3e22b4e589962ff7140e69aaa (table test-table [id=51cf1888c0b043a58dad08df4823144f]) successfully deleted
W20260812 05:23:35.883152 26680 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.31.100.1:37667 (error 108)
I20260812 05:23:35.883509 32144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0waFvT/build/release/bin/kudu with pid 27060
I20260812 05:23:35.883620 27062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:23:35.883490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.1:37667 (local address 127.0.0.1:40442)
0812 05:23:35.883542 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0812 05:23:35.883587 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.100.1:37667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260812 05:23:35.895267 26680 catalog_manager.cc:4584] TS 3f75ed97c1aa4752b59f6a1551419390 (127.31.100.1:37667): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 4caedac3e22b4e589962ff7140e69aaa: Network error: Client connection negotiation failed: client connection to 127.31.100.1:37667: connect: Connection refused (error 111)
I20260812 05:23:35.899103 32144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0waFvT/build/release/bin/kudu with pid 27302
I20260812 05:23:35.909763 32144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0waFvT/build/release/bin/kudu with pid 26654
I20260812 05:23:55.921481 32144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0waFvT/build/release/bin/kudu
/tmp/dist-test-task0waFvT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786512017613366-32144-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786512017613366-32144-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786512017613366-32144-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786512017613366-32144-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.100.62:37129
--webserver_interface=127.31.100.62
--webserver_port=43699
--keytab_file=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.62.keytab
--principal=kudu/127.31.100.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.100.20:45999
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.100.62:37129
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786512017613366-32144-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786512017613366-32144-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786512017613366-32144-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 05:23:56.002092 27590 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 05:23:56.002295 27590 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 05:23:56.002323 27590 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 05:23:56.003473 27590 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260812 05:23:56.003511 27590 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260812 05:23:56.003528 27590 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260812 05:23:56.003543 27590 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 05:23:56.003558 27590 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 05:23:56.003576 27590 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 05:23:56.003599 27590 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 05:23:56.003623 27590 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 05:23:56.005050 27590 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.100.20:45999
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786512017613366-32144-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786512017613366-32144-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.31.100.62:37129
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.62.keytab
--principal=kudu/127.31.100.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.100.62:37129
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786512017613366-32144-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.100.62
--webserver_port=43699
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27590
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786512017613366-32144-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 04:20:34 UTC on ccycloud.quasar-phqdqp.root.comops.site
build id 4581871
I20260812 05:23:56.005285 27590 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 05:23:56.005503 27590 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 05:23:56.010146 27596 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 05:23:56.010185 27600 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 05:23:56.010176 27605 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 05:23:56.010258 27590 server_base.cc:1034] running on GCE node
I20260812 05:23:56.010483 27590 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 05:23:56.010735 27590 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 05:23:56.011873 27590 hybrid_clock.cc:648] HybridClock initialized: now 1786512236011857 us; error 30 us; skew 500 ppm
Aug 12 05:23:56 dist-test-slave-42z9 krb5kdc[25677](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512236, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.100.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 05:23:56.017959 27590 init.cc:377] Logged in from keytab as kudu/127.31.100.62@KRBTEST.COM (short username kudu)
I20260812 05:23:56.019620 27590 webserver.cc:460] Webserver started at http://127.31.100.62:43699/ using document root <none> and password file <none>
I20260812 05:23:56.019812 27590 fs_manager.cc:359] Metadata directory not provided
I20260812 05:23:56.019860 27590 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 05:23:56.021106 27590 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260812 05:23:56.021741 27616 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 05:23:56.021888 27590 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260812 05:23:56.021981 27590 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786512017613366-32144-0/minicluster-data/master-0/data,/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786512017613366-32144-0/minicluster-data/master-0/wal
uuid: "a2ede0bc8af24780a89103af190035b8"
format_stamp: "Formatted at 2026-08-12 05:23:14 on dist-test-slave-42z9"
server_key: "a0b2f7fff55987d6788b7a426193d037"
server_key_iv: "bf518a542e138ca511cd0bbdc2024334"
server_key_version: "encryptionkey@0"
I20260812 05:23:56.022293 27590 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786512017613366-32144-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786512017613366-32144-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786512017613366-32144-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 05:23:56.033975 27590 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 05:23:56.035684 27590 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 05:23:56.035809 27590 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 05:23:56.039688 27590 rpc_server.cc:307] RPC server started. Bound to: 127.31.100.62:37129
I20260812 05:23:56.039727 27706 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.100.62:37129 every 8 connection(s)
I20260812 05:23:56.040076 27590 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786512017613366-32144-0/minicluster-data/master-0/data/info.pb
I20260812 05:23:56.041589 27709 sys_catalog.cc:263] Verifying existing consensus state
I20260812 05:23:56.042248 27709 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a2ede0bc8af24780a89103af190035b8: Bootstrap starting.
I20260812 05:23:56.047024 32144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0waFvT/build/release/bin/kudu as pid 27590
I20260812 05:23:56.047284 32144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0waFvT/build/release/bin/kudu
/tmp/dist-test-task0waFvT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786512017613366-32144-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786512017613366-32144-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786512017613366-32144-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786512017613366-32144-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.100.1:37667
--local_ip_for_outbound_sockets=127.31.100.1
--tserver_master_addrs=127.31.100.62:37129
--webserver_port=43255
--webserver_interface=127.31.100.1
--keytab_file=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.1.keytab
--principal=kudu/127.31.100.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.100.20:45999
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786512017613366-32144-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786512017613366-32144-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786512017613366-32144-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 05:23:56.050797 27709 log.cc:824] T 00000000000000000000000000000000 P a2ede0bc8af24780a89103af190035b8: Log is configured to *not* fsync() on all Append() calls
I20260812 05:23:56.052840 27709 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a2ede0bc8af24780a89103af190035b8: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 05:23:56.053102 27709 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a2ede0bc8af24780a89103af190035b8: Bootstrap complete.
I20260812 05:23:56.055203 27709 raft_consensus.cc:348] T 00000000000000000000000000000000 P a2ede0bc8af24780a89103af190035b8 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2ede0bc8af24780a89103af190035b8" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 37129 } }
I20260812 05:23:56.055425 27709 raft_consensus.cc:729] T 00000000000000000000000000000000 P a2ede0bc8af24780a89103af190035b8 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2ede0bc8af24780a89103af190035b8, State: Initialized, Role: FOLLOWER
I20260812 05:23:56.055581 27709 consensus_queue.cc:260] T 00000000000000000000000000000000 P a2ede0bc8af24780a89103af190035b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2ede0bc8af24780a89103af190035b8" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 37129 } }
I20260812 05:23:56.055642 27709 raft_consensus.cc:388] T 00000000000000000000000000000000 P a2ede0bc8af24780a89103af190035b8 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 05:23:56.055688 27709 raft_consensus.cc:482] T 00000000000000000000000000000000 P a2ede0bc8af24780a89103af190035b8 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 05:23:56.055733 27709 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a2ede0bc8af24780a89103af190035b8 [term 2 FOLLOWER]: Advancing to term 3
I20260812 05:23:56.056705 27709 raft_consensus.cc:504] T 00000000000000000000000000000000 P a2ede0bc8af24780a89103af190035b8 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2ede0bc8af24780a89103af190035b8" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 37129 } }
I20260812 05:23:56.056803 27709 leader_election.cc:302] T 00000000000000000000000000000000 P a2ede0bc8af24780a89103af190035b8 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a2ede0bc8af24780a89103af190035b8; no voters:
I20260812 05:23:56.056957 27709 leader_election.cc:288] T 00000000000000000000000000000000 P a2ede0bc8af24780a89103af190035b8 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260812 05:23:56.057013 27713 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a2ede0bc8af24780a89103af190035b8 [term 3 FOLLOWER]: Leader election won for term 3
I20260812 05:23:56.057176 27713 raft_consensus.cc:686] T 00000000000000000000000000000000 P a2ede0bc8af24780a89103af190035b8 [term 3 LEADER]: Becoming Leader. State: Replica: a2ede0bc8af24780a89103af190035b8, State: Running, Role: LEADER
I20260812 05:23:56.057214 27709 sys_catalog.cc:564] T 00000000000000000000000000000000 P a2ede0bc8af24780a89103af190035b8 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 05:23:56.057267 27713 consensus_queue.cc:237] T 00000000000000000000000000000000 P a2ede0bc8af24780a89103af190035b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2ede0bc8af24780a89103af190035b8" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 37129 } }
I20260812 05:23:56.057717 27714 sys_catalog.cc:455] T 00000000000000000000000000000000 P a2ede0bc8af24780a89103af190035b8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "a2ede0bc8af24780a89103af190035b8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2ede0bc8af24780a89103af190035b8" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 37129 } } }
I20260812 05:23:56.057811 27714 sys_catalog.cc:458] T 00000000000000000000000000000000 P a2ede0bc8af24780a89103af190035b8 [sys.catalog]: This master's current role is: LEADER
I20260812 05:23:56.058028 27715 sys_catalog.cc:455] T 00000000000000000000000000000000 P a2ede0bc8af24780a89103af190035b8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a2ede0bc8af24780a89103af190035b8. Latest consensus state: current_term: 3 leader_uuid: "a2ede0bc8af24780a89103af190035b8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2ede0bc8af24780a89103af190035b8" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 37129 } } }
I20260812 05:23:56.058159 27715 sys_catalog.cc:458] T 00000000000000000000000000000000 P a2ede0bc8af24780a89103af190035b8 [sys.catalog]: This master's current role is: LEADER
I20260812 05:23:56.058164 27722 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 05:23:56.058964 27722 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 05:23:56.059073 27722 catalog_manager.cc:1259] Loaded cluster ID: 946abde4098a48769bf5aba02bbd3087
I20260812 05:23:56.059154 27722 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 05:23:56.060612 27722 catalog_manager.cc:1504] Loading token signing keys...
I20260812 05:23:56.060801 27722 catalog_manager.cc:5879] T 00000000000000000000000000000000 P a2ede0bc8af24780a89103af190035b8: Loaded TSK: 0
I20260812 05:23:56.060843 27722 catalog_manager.cc:5879] T 00000000000000000000000000000000 P a2ede0bc8af24780a89103af190035b8: Loaded TSK: 1
F20260812 05:23:56.061080 27722 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fdff16629ad google::LogMessage::Fail() at ??:0
@ 0x7fdff166588a google::LogMessage::SendToLog() at ??:0
@ 0x7fdff16624de google::LogMessage::Flush() at ??:0
@ 0x7fdff1663a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fdff3dcf104 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7fdff3dccd41 kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7fdff95bc1cd kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7fdff95bc48e kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7fdff95bc594 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7fdff9582c48 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7fdff958cad0 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fdff35300d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fdff3525954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fdff43896db start_thread at ??:0
@ 0x7fdff016061f clone at ??:0
W20260812 05:23:56.148237 27711 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 05:23:56.148439 27711 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 05:23:56.148461 27711 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 05:23:56.149652 27711 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 05:23:56.149698 27711 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 05:23:56.149727 27711 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.100.1
I20260812 05:23:56.151224 27711 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.100.20:45999
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786512017613366-32144-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786512017613366-32144-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.1.keytab
--principal=kudu/127.31.100.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.100.1:37667
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786512017613366-32144-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.100.1
--webserver_port=43255
--tserver_master_addrs=127.31.100.62:37129
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27711
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.100.1
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786512017613366-32144-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 04:20:34 UTC on ccycloud.quasar-phqdqp.root.comops.site
build id 4581871
I20260812 05:23:56.151443 27711 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 05:23:56.151664 27711 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 05:23:56.156598 27777 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 05:23:56.156594 27778 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 05:23:56.156673 27782 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 05:23:56.157212 27711 server_base.cc:1034] running on GCE node
I20260812 05:23:56.157399 27711 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 05:23:56.157653 27711 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 05:23:56.158797 27711 hybrid_clock.cc:648] HybridClock initialized: now 1786512236158783 us; error 31 us; skew 500 ppm
Aug 12 05:23:56 dist-test-slave-42z9 krb5kdc[25677](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512236, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.100.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 05:23:56.161386 27711 init.cc:377] Logged in from keytab as kudu/127.31.100.1@KRBTEST.COM (short username kudu)
I20260812 05:23:56.161859 27711 webserver.cc:460] Webserver started at http://127.31.100.1:43255/ using document root <none> and password file <none>
I20260812 05:23:56.162061 27711 fs_manager.cc:359] Metadata directory not provided
I20260812 05:23:56.162111 27711 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 05:23:56.163483 27711 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260812 05:23:56.164168 27797 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 05:23:56.164324 27711 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260812 05:23:56.164408 27711 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786512017613366-32144-0/minicluster-data/ts-0/data,/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786512017613366-32144-0/minicluster-data/ts-0/wal
uuid: "3f75ed97c1aa4752b59f6a1551419390"
format_stamp: "Formatted at 2026-08-12 05:23:14 on dist-test-slave-42z9"
server_key: "1ff21222bd34bb43ebd41add3e2d3a43"
server_key_iv: "a68b8fa533bdffcbc4072c9da796609f"
server_key_version: "encryptionkey@0"
I20260812 05:23:56.164737 27711 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786512017613366-32144-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786512017613366-32144-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786512017613366-32144-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 05:23:56.176203 27711 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 05:23:56.178184 27711 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 05:23:56.178313 27711 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 05:23:56.178514 27711 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 05:23:56.178931 27811 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260812 05:23:56.180847 27711 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260812 05:23:56.180889 27711 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260812 05:23:56.180922 27711 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260812 05:23:56.181586 27711 ts_tablet_manager.cc:610] Registered 1 tablets
I20260812 05:23:56.181620 27711 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.001s sys 0.000s
I20260812 05:23:56.181710 27811 tablet_bootstrap.cc:492] T 4caedac3e22b4e589962ff7140e69aaa P 3f75ed97c1aa4752b59f6a1551419390: Bootstrap starting.
I20260812 05:23:56.188174 27711 rpc_server.cc:307] RPC server started. Bound to: 127.31.100.1:37667
I20260812 05:23:56.188232 28001 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.100.1:37667 every 8 connection(s)
I20260812 05:23:56.188575 27711 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786512017613366-32144-0/minicluster-data/ts-0/data/info.pb
I20260812 05:23:56.193006 32144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0waFvT/build/release/bin/kudu as pid 27711
I20260812 05:23:56.193286 32144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0waFvT/build/release/bin/kudu
/tmp/dist-test-task0waFvT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786512017613366-32144-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786512017613366-32144-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786512017613366-32144-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786512017613366-32144-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.100.2:39191
--local_ip_for_outbound_sockets=127.31.100.2
--tserver_master_addrs=127.31.100.62:37129
--webserver_port=40853
--webserver_interface=127.31.100.2
--keytab_file=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.2.keytab
--principal=kudu/127.31.100.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.100.20:45999
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786512017613366-32144-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786512017613366-32144-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786512017613366-32144-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 05:23:56.193641 27811 log.cc:824] T 4caedac3e22b4e589962ff7140e69aaa P 3f75ed97c1aa4752b59f6a1551419390: Log is configured to *not* fsync() on all Append() calls
I20260812 05:23:56.196310 27811 tablet_bootstrap.cc:492] T 4caedac3e22b4e589962ff7140e69aaa P 3f75ed97c1aa4752b59f6a1551419390: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 05:23:56.196630 27811 tablet_bootstrap.cc:492] T 4caedac3e22b4e589962ff7140e69aaa P 3f75ed97c1aa4752b59f6a1551419390: Bootstrap complete.
I20260812 05:23:56.197786 27811 ts_tablet_manager.cc:1397] T 4caedac3e22b4e589962ff7140e69aaa P 3f75ed97c1aa4752b59f6a1551419390: Time spent bootstrapping tablet: real 0.016s user 0.000s sys 0.014s
I20260812 05:23:56.199206 27811 raft_consensus.cc:348] T 4caedac3e22b4e589962ff7140e69aaa P 3f75ed97c1aa4752b59f6a1551419390 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df8db312fb6c4c57aa51f490c25fa09f" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 45871 } } peers { permanent_uuid: "edf25888a7e648348650523730de9401" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 39191 } } peers { permanent_uuid: "3f75ed97c1aa4752b59f6a1551419390" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 37667 } }
I20260812 05:23:56.199568 27811 raft_consensus.cc:729] T 4caedac3e22b4e589962ff7140e69aaa P 3f75ed97c1aa4752b59f6a1551419390 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f75ed97c1aa4752b59f6a1551419390, State: Initialized, Role: FOLLOWER
I20260812 05:23:56.199753 27811 consensus_queue.cc:260] T 4caedac3e22b4e589962ff7140e69aaa P 3f75ed97c1aa4752b59f6a1551419390 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df8db312fb6c4c57aa51f490c25fa09f" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 45871 } } peers { permanent_uuid: "edf25888a7e648348650523730de9401" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 39191 } } peers { permanent_uuid: "3f75ed97c1aa4752b59f6a1551419390" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 37667 } }
I20260812 05:23:56.200049 27811 ts_tablet_manager.cc:1428] T 4caedac3e22b4e589962ff7140e69aaa P 3f75ed97c1aa4752b59f6a1551419390: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
W20260812 05:23:56.257203 28006 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 05:23:56.189207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.62:37129 (local address 127.31.100.1:35867)
0812 05:23:56.189371 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0812 05:23:56.189412 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0812 05:23:56.189861 (+ 449us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:23:56.256969 (+ 67108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.100.62:37129: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":77,"threads_started":1}
W20260812 05:23:56.257490 28002 heartbeater.cc:644] Failed to heartbeat to 127.31.100.62:37129 (0 consecutive failures): Network error: Failed to ping master at 127.31.100.62:37129: Client connection negotiation failed: client connection to 127.31.100.62:37129: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260812 05:23:56.257846 28006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:23:56.257722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.62:37129 (local address 127.31.100.1:54457)
0812 05:23:56.257776 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0812 05:23:56.257822 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.100.62:37129: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260812 05:23:56.258183 28006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:23:56.258097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.62:37129 (local address 127.31.100.1:53825)
0812 05:23:56.258138 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0812 05:23:56.258165 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.100.62:37129: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260812 05:23:56.258293 28002 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260812 05:23:56.275314 28009 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 05:23:56.275496 28009 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 05:23:56.275516 28009 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 05:23:56.276610 28009 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 05:23:56.276649 28009 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 05:23:56.276677 28009 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.100.2
I20260812 05:23:56.278115 28009 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.100.20:45999
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786512017613366-32144-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786512017613366-32144-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.2.keytab
--principal=kudu/127.31.100.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.100.2:39191
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786512017613366-32144-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.100.2
--webserver_port=40853
--tserver_master_addrs=127.31.100.62:37129
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28009
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.100.2
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786512017613366-32144-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 04:20:34 UTC on ccycloud.quasar-phqdqp.root.comops.site
build id 4581871
I20260812 05:23:56.278311 28009 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 05:23:56.278505 28009 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 05:23:56.283174 28023 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 05:23:56.283181 28025 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 05:23:56.283315 28009 server_base.cc:1034] running on GCE node
W20260812 05:23:56.283349 28027 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 05:23:56.283578 28009 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 05:23:56.283772 28009 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 05:23:56.284904 28009 hybrid_clock.cc:648] HybridClock initialized: now 1786512236284891 us; error 28 us; skew 500 ppm
Aug 12 05:23:56 dist-test-slave-42z9 krb5kdc[25677](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512236, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.100.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 05:23:56.287240 28009 init.cc:377] Logged in from keytab as kudu/127.31.100.2@KRBTEST.COM (short username kudu)
I20260812 05:23:56.287647 28009 webserver.cc:460] Webserver started at http://127.31.100.2:40853/ using document root <none> and password file <none>
I20260812 05:23:56.287832 28009 fs_manager.cc:359] Metadata directory not provided
I20260812 05:23:56.287875 28009 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 05:23:56.289182 28009 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260812 05:23:56.289754 28041 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 05:23:56.289891 28009 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260812 05:23:56.289973 28009 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786512017613366-32144-0/minicluster-data/ts-1/data,/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786512017613366-32144-0/minicluster-data/ts-1/wal
uuid: "edf25888a7e648348650523730de9401"
format_stamp: "Formatted at 2026-08-12 05:23:14 on dist-test-slave-42z9"
server_key: "e08482d6fcbcc44b05a705170d41aa7f"
server_key_iv: "f08036fffbeee9c4ac9bcf06ad123959"
server_key_version: "encryptionkey@0"
I20260812 05:23:56.290278 28009 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786512017613366-32144-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786512017613366-32144-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786512017613366-32144-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 05:23:56.305879 28009 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 05:23:56.307526 28009 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 05:23:56.307658 28009 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 05:23:56.307847 28009 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 05:23:56.308086 28009 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 05:23:56.308116 28009 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 05:23:56.308135 28009 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 05:23:56.308151 28009 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 05:23:56.313830 28009 rpc_server.cc:307] RPC server started. Bound to: 127.31.100.2:39191
I20260812 05:23:56.313885 28251 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.100.2:39191 every 8 connection(s)
I20260812 05:23:56.314182 28009 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786512017613366-32144-0/minicluster-data/ts-1/data/info.pb
I20260812 05:23:56.315135 28259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:23:56.314775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.62:37129 (local address 127.31.100.2:48647)
0812 05:23:56.314932 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0812 05:23:56.314995 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.100.62:37129: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":91,"threads_started":1}
W20260812 05:23:56.315513 28256 heartbeater.cc:644] Failed to heartbeat to 127.31.100.62:37129 (0 consecutive failures): Network error: Failed to ping master at 127.31.100.62:37129: Client connection negotiation failed: client connection to 127.31.100.62:37129: connect: Connection refused (error 111)
I20260812 05:23:56.315749 28259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:23:56.315673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.62:37129 (local address 127.31.100.2:57809)
0812 05:23:56.315708 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0812 05:23:56.315733 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.100.62:37129: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
I20260812 05:23:56.315935 28259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:23:56.315891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.62:37129 (local address 127.31.100.2:34619)
0812 05:23:56.315910 (+ 19us) negotiation.cc:107] Waiting for socket to connect
0812 05:23:56.315926 (+ 16us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.100.62:37129: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":11}
W20260812 05:23:56.316013 28256 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260812 05:23:56.323719 32144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0waFvT/build/release/bin/kudu as pid 28009
I20260812 05:23:56.323962 32144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0waFvT/build/release/bin/kudu
/tmp/dist-test-task0waFvT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786512017613366-32144-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786512017613366-32144-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786512017613366-32144-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786512017613366-32144-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.100.3:45871
--local_ip_for_outbound_sockets=127.31.100.3
--tserver_master_addrs=127.31.100.62:37129
--webserver_port=38899
--webserver_interface=127.31.100.3
--keytab_file=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.3.keytab
--principal=kudu/127.31.100.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.100.20:45999
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786512017613366-32144-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786512017613366-32144-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786512017613366-32144-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 05:23:56.405112 28261 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 05:23:56.405300 28261 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 05:23:56.405321 28261 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 05:23:56.406394 28261 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 05:23:56.406428 28261 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 05:23:56.406452 28261 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.100.3
I20260812 05:23:56.407859 28261 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.100.20:45999
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786512017613366-32144-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786512017613366-32144-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.3.keytab
--principal=kudu/127.31.100.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.100.3:45871
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786512017613366-32144-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.100.3
--webserver_port=38899
--tserver_master_addrs=127.31.100.62:37129
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28261
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.100.3
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786512017613366-32144-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 04:20:34 UTC on ccycloud.quasar-phqdqp.root.comops.site
build id 4581871
I20260812 05:23:56.408033 28261 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 05:23:56.408197 28261 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 05:23:56.412773 28273 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 05:23:56.412829 28272 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 05:23:56.412854 28277 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 05:23:56.413041 28261 server_base.cc:1034] running on GCE node
I20260812 05:23:56.413228 28261 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 05:23:56.413406 28261 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 05:23:56.414558 28261 hybrid_clock.cc:648] HybridClock initialized: now 1786512236414541 us; error 36 us; skew 500 ppm
Aug 12 05:23:56 dist-test-slave-42z9 krb5kdc[25677](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512236, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.100.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 05:23:56.416718 28261 init.cc:377] Logged in from keytab as kudu/127.31.100.3@KRBTEST.COM (short username kudu)
I20260812 05:23:56.417066 28261 webserver.cc:460] Webserver started at http://127.31.100.3:38899/ using document root <none> and password file <none>
I20260812 05:23:56.417225 28261 fs_manager.cc:359] Metadata directory not provided
I20260812 05:23:56.417263 28261 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 05:23:56.418458 28261 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260812 05:23:56.418974 28292 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 05:23:56.419126 28261 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260812 05:23:56.419209 28261 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786512017613366-32144-0/minicluster-data/ts-2/data,/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786512017613366-32144-0/minicluster-data/ts-2/wal
uuid: "df8db312fb6c4c57aa51f490c25fa09f"
format_stamp: "Formatted at 2026-08-12 05:23:14 on dist-test-slave-42z9"
server_key: "9ce08b8ccb671a16de628348a0a75d2a"
server_key_iv: "186a8874839af3036fc47945c19039fb"
server_key_version: "encryptionkey@0"
I20260812 05:23:56.419494 28261 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786512017613366-32144-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786512017613366-32144-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786512017613366-32144-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 05:23:56.445189 28261 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 05:23:56.446834 28261 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 05:23:56.446941 28261 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 05:23:56.447125 28261 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 05:23:56.447366 28261 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 05:23:56.447396 28261 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 05:23:56.447415 28261 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 05:23:56.447430 28261 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 05:23:56.452729 28261 rpc_server.cc:307] RPC server started. Bound to: 127.31.100.3:45871
I20260812 05:23:56.452793 28494 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.100.3:45871 every 8 connection(s)
I20260812 05:23:56.453084 28261 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786512017613366-32144-0/minicluster-data/ts-2/data/info.pb
I20260812 05:23:56.454044 28504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:23:56.453701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.62:37129 (local address 127.31.100.3:37379)
0812 05:23:56.453843 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0812 05:23:56.453905 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.100.62:37129: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":72,"threads_started":1}
W20260812 05:23:56.454397 28495 heartbeater.cc:644] Failed to heartbeat to 127.31.100.62:37129 (0 consecutive failures): Network error: Failed to ping master at 127.31.100.62:37129: Client connection negotiation failed: client connection to 127.31.100.62:37129: connect: Connection refused (error 111)
I20260812 05:23:56.454639 28504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:23:56.454560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.62:37129 (local address 127.31.100.3:34627)
0812 05:23:56.454597 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0812 05:23:56.454622 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.100.62:37129: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
I20260812 05:23:56.454840 28504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:23:56.454782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.62:37129 (local address 127.31.100.3:49527)
0812 05:23:56.454811 (+ 29us) negotiation.cc:107] Waiting for socket to connect
0812 05:23:56.454828 (+ 17us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.100.62:37129: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":20}
W20260812 05:23:56.454919 28495 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260812 05:23:56.459534 32144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0waFvT/build/release/bin/kudu as pid 28261
I20260812 05:23:57.258800 28508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:23:57.258588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.62:37129 (local address 127.31.100.1:40619)
0812 05:23:57.258701 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0812 05:23:57.258734 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.100.62:37129: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":62,"threads_started":1}
I20260812 05:23:57.316473 28511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:23:57.316284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.62:37129 (local address 127.31.100.2:51823)
0812 05:23:57.316389 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0812 05:23:57.316421 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.100.62:37129: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92,"thread_start_us":54,"threads_started":1}
I20260812 05:23:57.455382 28512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:23:57.455193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.62:37129 (local address 127.31.100.3:56813)
0812 05:23:57.455304 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0812 05:23:57.455335 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.100.62:37129: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":60,"threads_started":1}
I20260812 05:23:57.479800 28513 raft_consensus.cc:482] T 4caedac3e22b4e589962ff7140e69aaa P 3f75ed97c1aa4752b59f6a1551419390 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 05:23:57.479939 28513 raft_consensus.cc:504] T 4caedac3e22b4e589962ff7140e69aaa P 3f75ed97c1aa4752b59f6a1551419390 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df8db312fb6c4c57aa51f490c25fa09f" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 45871 } } peers { permanent_uuid: "edf25888a7e648348650523730de9401" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 39191 } } peers { permanent_uuid: "3f75ed97c1aa4752b59f6a1551419390" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 37667 } }
I20260812 05:23:57.480248 28513 leader_election.cc:288] T 4caedac3e22b4e589962ff7140e69aaa P 3f75ed97c1aa4752b59f6a1551419390 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers df8db312fb6c4c57aa51f490c25fa09f (127.31.100.3:45871), edf25888a7e648348650523730de9401 (127.31.100.2:39191)
Aug 12 05:23:57 dist-test-slave-42z9 krb5kdc[25677](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512236, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.100.1@KRBTEST.COM for kudu/127.31.100.2@KRBTEST.COM
Aug 12 05:23:57 dist-test-slave-42z9 krb5kdc[25677](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512236, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.100.1@KRBTEST.COM for kudu/127.31.100.3@KRBTEST.COM
I20260812 05:23:57.491127 28519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:23:57.480450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.2:39191 (local address 127.31.100.1:44685)
0812 05:23:57.480563 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0812 05:23:57.480573 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0812 05:23:57.480623 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:23:57.482278 (+ 1655us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:23:57.482279 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:23:57.482399 (+ 120us) client_negotiation.cc:189] Negotiated authn=SASL
0812 05:23:57.482433 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:23:57.482434 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:23:57.483944 (+ 1510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:23:57.483945 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:23:57.484707 (+ 762us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:23:57.484710 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:23:57.485273 (+ 563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:23:57.485274 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:23:57.485384 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:23:57.485914 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 05:23:57.485922 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0812 05:23:57.487284 (+ 1362us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 05:23:57.489204 (+ 1920us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 05:23:57.489211 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 05:23:57.489213 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 05:23:57.489612 (+ 399us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 05:23:57.489936 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 05:23:57.489937 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 05:23:57.489937 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 05:23:57.490109 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 05:23:57.490586 (+ 477us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 05:23:57.490587 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 05:23:57.490718 (+ 131us) client_negotiation.cc:769] Sending connection context
0812 05:23:57.490831 (+ 113us) client_negotiation.cc:240] Negotiation successful
0812 05:23:57.491018 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100,"mutex_wait_us":73,"thread_start_us":62,"threads_started":1}
I20260812 05:23:57.491155 28522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:23:57.480545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.100.1:46957 (local address 127.31.100.3:45871)
0812 05:23:57.480679 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0812 05:23:57.480691 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0812 05:23:57.480725 (+ 34us) server_negotiation.cc:407] Connection header received
0812 05:23:57.481305 (+ 580us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:23:57.481308 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:23:57.481363 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:23:57.481524 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:23:57.482303 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:23:57.483816 (+ 1513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:23:57.484784 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:23:57.485191 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:23:57.485289 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:23:57.487744 (+ 2455us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 05:23:57.487748 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 05:23:57.487748 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 05:23:57.487760 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0812 05:23:57.489274 (+ 1514us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:23:57.489710 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:23:57.489712 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:23:57.489712 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:23:57.489831 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:23:57.490180 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:23:57.490181 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:23:57.490181 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:23:57.490494 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 05:23:57.490579 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0812 05:23:57.490903 (+ 324us) server_negotiation.cc:299] Negotiation successful
0812 05:23:57.491045 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94,"thread_start_us":58,"threads_started":1}
I20260812 05:23:57.491153 28524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:23:57.480596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.100.1:44685 (local address 127.31.100.2:39191)
0812 05:23:57.480736 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0812 05:23:57.480743 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 05:23:57.480792 (+ 49us) server_negotiation.cc:407] Connection header received
0812 05:23:57.481304 (+ 512us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:23:57.481308 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:23:57.481363 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:23:57.481523 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:23:57.482520 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:23:57.483805 (+ 1285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:23:57.484802 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:23:57.485192 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:23:57.485289 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:23:57.487385 (+ 2096us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 05:23:57.487390 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 05:23:57.487390 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 05:23:57.487403 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0812 05:23:57.489086 (+ 1683us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:23:57.489696 (+ 610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:23:57.489699 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:23:57.489699 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:23:57.489831 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:23:57.490180 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:23:57.490181 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:23:57.490181 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:23:57.490495 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 05:23:57.490597 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0812 05:23:57.490902 (+ 305us) server_negotiation.cc:299] Negotiation successful
0812 05:23:57.491037 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100,"thread_start_us":53,"threads_started":1}
I20260812 05:23:57.491190 28508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:23:57.480373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.3:45871 (local address 127.31.100.1:46957)
0812 05:23:57.480425 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0812 05:23:57.480440 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 05:23:57.480508 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:23:57.481701 (+ 1193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:23:57.481703 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:23:57.481990 (+ 287us) client_negotiation.cc:189] Negotiated authn=SASL
0812 05:23:57.482196 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:23:57.482202 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:23:57.483895 (+ 1693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:23:57.483896 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:23:57.484708 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:23:57.484710 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:23:57.485282 (+ 572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:23:57.485283 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:23:57.485383 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:23:57.485952 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 05:23:57.485957 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0812 05:23:57.487655 (+ 1698us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 05:23:57.489347 (+ 1692us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 05:23:57.489348 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 05:23:57.489349 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 05:23:57.489626 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 05:23:57.489933 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 05:23:57.489934 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 05:23:57.489934 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 05:23:57.490109 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 05:23:57.490562 (+ 453us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 05:23:57.490567 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 05:23:57.490718 (+ 151us) client_negotiation.cc:769] Sending connection context
0812 05:23:57.490846 (+ 128us) client_negotiation.cc:240] Negotiation successful
0812 05:23:57.491085 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30,"mutex_wait_us":109}
W20260812 05:23:57.492432 27798 leader_election.cc:341] T 4caedac3e22b4e589962ff7140e69aaa P 3f75ed97c1aa4752b59f6a1551419390 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer df8db312fb6c4c57aa51f490c25fa09f (127.31.100.3:45871): Not found: Tablet not found: 4caedac3e22b4e589962ff7140e69aaa
W20260812 05:23:57.492507 27798 leader_election.cc:341] T 4caedac3e22b4e589962ff7140e69aaa P 3f75ed97c1aa4752b59f6a1551419390 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer edf25888a7e648348650523730de9401 (127.31.100.2:39191): Not found: Tablet not found: 4caedac3e22b4e589962ff7140e69aaa
I20260812 05:23:57.492540 27798 leader_election.cc:302] T 4caedac3e22b4e589962ff7140e69aaa P 3f75ed97c1aa4752b59f6a1551419390 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f75ed97c1aa4752b59f6a1551419390; no voters: df8db312fb6c4c57aa51f490c25fa09f, edf25888a7e648348650523730de9401
I20260812 05:23:57.492646 28513 raft_consensus.cc:2726] T 4caedac3e22b4e589962ff7140e69aaa P 3f75ed97c1aa4752b59f6a1551419390 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260812 05:23:58.259464 28525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:23:58.259223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.62:37129 (local address 127.31.100.1:33385)
0812 05:23:58.259377 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0812 05:23:58.259417 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.100.62:37129: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":65,"threads_started":1}
I20260812 05:23:58.317039 28526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:23:58.316836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.62:37129 (local address 127.31.100.2:38539)
0812 05:23:58.316967 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0812 05:23:58.317001 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.100.62:37129: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":70,"threads_started":1}
I20260812 05:23:58.455998 28527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:23:58.455742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.62:37129 (local address 127.31.100.3:57947)
0812 05:23:58.455887 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0812 05:23:58.455954 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.100.62:37129: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":71,"threads_started":1}
I20260812 05:23:59.249927 28528 raft_consensus.cc:482] T 4caedac3e22b4e589962ff7140e69aaa P 3f75ed97c1aa4752b59f6a1551419390 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 05:23:59.250025 28528 raft_consensus.cc:504] T 4caedac3e22b4e589962ff7140e69aaa P 3f75ed97c1aa4752b59f6a1551419390 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df8db312fb6c4c57aa51f490c25fa09f" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 45871 } } peers { permanent_uuid: "edf25888a7e648348650523730de9401" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 39191 } } peers { permanent_uuid: "3f75ed97c1aa4752b59f6a1551419390" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 37667 } }
I20260812 05:23:59.250181 28528 leader_election.cc:288] T 4caedac3e22b4e589962ff7140e69aaa P 3f75ed97c1aa4752b59f6a1551419390 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers df8db312fb6c4c57aa51f490c25fa09f (127.31.100.3:45871), edf25888a7e648348650523730de9401 (127.31.100.2:39191)
W20260812 05:23:59.250583 27798 leader_election.cc:341] T 4caedac3e22b4e589962ff7140e69aaa P 3f75ed97c1aa4752b59f6a1551419390 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer edf25888a7e648348650523730de9401 (127.31.100.2:39191): Not found: Tablet not found: 4caedac3e22b4e589962ff7140e69aaa
W20260812 05:23:59.250653 27798 leader_election.cc:341] T 4caedac3e22b4e589962ff7140e69aaa P 3f75ed97c1aa4752b59f6a1551419390 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer df8db312fb6c4c57aa51f490c25fa09f (127.31.100.3:45871): Not found: Tablet not found: 4caedac3e22b4e589962ff7140e69aaa
I20260812 05:23:59.250702 27798 leader_election.cc:302] T 4caedac3e22b4e589962ff7140e69aaa P 3f75ed97c1aa4752b59f6a1551419390 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f75ed97c1aa4752b59f6a1551419390; no voters: df8db312fb6c4c57aa51f490c25fa09f, edf25888a7e648348650523730de9401
I20260812 05:23:59.250763 28528 raft_consensus.cc:2726] T 4caedac3e22b4e589962ff7140e69aaa P 3f75ed97c1aa4752b59f6a1551419390 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260812 05:23:59.260027 28529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:23:59.259850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.62:37129 (local address 127.31.100.1:53451)
0812 05:23:59.259951 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0812 05:23:59.259982 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.100.62:37129: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86,"thread_start_us":54,"threads_started":1}
I20260812 05:23:59.317608 28530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:23:59.317417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.62:37129 (local address 127.31.100.2:50813)
0812 05:23:59.317545 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0812 05:23:59.317574 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.100.62:37129: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":79,"threads_started":1}
I20260812 05:23:59.456619 28533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:23:59.456407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.62:37129 (local address 127.31.100.3:55759)
0812 05:23:59.456544 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0812 05:23:59.456578 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.100.62:37129: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":65,"threads_started":1}
I20260812 05:24:00.260637 28536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:24:00.260419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.62:37129 (local address 127.31.100.1:57013)
0812 05:24:00.260552 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0812 05:24:00.260591 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.100.62:37129: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":61,"threads_started":1}
I20260812 05:24:00.318111 28538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:24:00.317943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.62:37129 (local address 127.31.100.2:41983)
0812 05:24:00.318042 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0812 05:24:00.318073 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.100.62:37129: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85,"thread_start_us":54,"threads_started":1}
I20260812 05:24:00.457240 28539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:24:00.457045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.62:37129 (local address 127.31.100.3:40397)
0812 05:24:00.457158 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0812 05:24:00.457196 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.100.62:37129: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":60,"threads_started":1}
I20260812 05:24:01.097500 28542 raft_consensus.cc:482] T 4caedac3e22b4e589962ff7140e69aaa P 3f75ed97c1aa4752b59f6a1551419390 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 05:24:01.097599 28542 raft_consensus.cc:504] T 4caedac3e22b4e589962ff7140e69aaa P 3f75ed97c1aa4752b59f6a1551419390 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df8db312fb6c4c57aa51f490c25fa09f" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 45871 } } peers { permanent_uuid: "edf25888a7e648348650523730de9401" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 39191 } } peers { permanent_uuid: "3f75ed97c1aa4752b59f6a1551419390" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 37667 } }
I20260812 05:24:01.097736 28542 leader_election.cc:288] T 4caedac3e22b4e589962ff7140e69aaa P 3f75ed97c1aa4752b59f6a1551419390 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers df8db312fb6c4c57aa51f490c25fa09f (127.31.100.3:45871), edf25888a7e648348650523730de9401 (127.31.100.2:39191)
W20260812 05:24:01.098124 27798 leader_election.cc:341] T 4caedac3e22b4e589962ff7140e69aaa P 3f75ed97c1aa4752b59f6a1551419390 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer edf25888a7e648348650523730de9401 (127.31.100.2:39191): Not found: Tablet not found: 4caedac3e22b4e589962ff7140e69aaa
W20260812 05:24:01.098199 27798 leader_election.cc:341] T 4caedac3e22b4e589962ff7140e69aaa P 3f75ed97c1aa4752b59f6a1551419390 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer df8db312fb6c4c57aa51f490c25fa09f (127.31.100.3:45871): Not found: Tablet not found: 4caedac3e22b4e589962ff7140e69aaa
I20260812 05:24:01.098225 27798 leader_election.cc:302] T 4caedac3e22b4e589962ff7140e69aaa P 3f75ed97c1aa4752b59f6a1551419390 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f75ed97c1aa4752b59f6a1551419390; no voters: df8db312fb6c4c57aa51f490c25fa09f, edf25888a7e648348650523730de9401
I20260812 05:24:01.098309 28542 raft_consensus.cc:2726] T 4caedac3e22b4e589962ff7140e69aaa P 3f75ed97c1aa4752b59f6a1551419390 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260812 05:24:01.261318 28544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:24:01.261096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.62:37129 (local address 127.31.100.1:43293)
0812 05:24:01.261235 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0812 05:24:01.261271 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.100.62:37129: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":64,"threads_started":1}
I20260812 05:24:01.318657 28545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:24:01.318475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.62:37129 (local address 127.31.100.2:53293)
0812 05:24:01.318585 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0812 05:24:01.318616 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.100.62:37129: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":59,"threads_started":1}
I20260812 05:24:01.457863 28546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:24:01.457658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.62:37129 (local address 127.31.100.3:57143)
0812 05:24:01.457786 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0812 05:24:01.457822 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.100.62:37129: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":72,"threads_started":1}
I20260812 05:24:02.261981 28547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:24:02.261733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.62:37129 (local address 127.31.100.1:33745)
0812 05:24:02.261891 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0812 05:24:02.261932 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.100.62:37129: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":62,"threads_started":1}
I20260812 05:24:02.319275 28549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:24:02.319067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.62:37129 (local address 127.31.100.2:35425)
0812 05:24:02.319195 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0812 05:24:02.319234 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.100.62:37129: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":61,"threads_started":1}
I20260812 05:24:02.458467 28551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:24:02.458276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.62:37129 (local address 127.31.100.3:43373)
0812 05:24:02.458390 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0812 05:24:02.458426 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.100.62:37129: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":62,"threads_started":1}
I20260812 05:24:03.262588 28552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:24:03.262384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.62:37129 (local address 127.31.100.1:59409)
0812 05:24:03.262504 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0812 05:24:03.262542 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.100.62:37129: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":62,"threads_started":1}
I20260812 05:24:03.319888 28553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:24:03.319701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.62:37129 (local address 127.31.100.2:37739)
0812 05:24:03.319815 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0812 05:24:03.319848 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.100.62:37129: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":66,"threads_started":1}
I20260812 05:24:03.459072 28554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:24:03.458881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.62:37129 (local address 127.31.100.3:51531)
0812 05:24:03.458990 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0812 05:24:03.459023 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.100.62:37129: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":61,"threads_started":1}
I20260812 05:24:04.263244 28556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:24:04.263006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.62:37129 (local address 127.31.100.1:47205)
0812 05:24:04.263158 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0812 05:24:04.263199 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.100.62:37129: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":62,"threads_started":1}
I20260812 05:24:04.320533 28557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:24:04.320288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.62:37129 (local address 127.31.100.2:45991)
0812 05:24:04.320442 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0812 05:24:04.320485 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.100.62:37129: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":81,"threads_started":1}
I20260812 05:24:04.459728 28558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:24:04.459504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.62:37129 (local address 127.31.100.3:38119)
0812 05:24:04.459649 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0812 05:24:04.459685 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.100.62:37129: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":79,"threads_started":1}
I20260812 05:24:05.264019 28559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:24:05.263771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.62:37129 (local address 127.31.100.1:42477)
0812 05:24:05.263925 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0812 05:24:05.263968 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.100.62:37129: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":78,"threads_started":1}
I20260812 05:24:05.321130 28560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:24:05.320919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.62:37129 (local address 127.31.100.2:45517)
0812 05:24:05.321052 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0812 05:24:05.321088 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.100.62:37129: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":56,"threads_started":1}
I20260812 05:24:05.460364 28562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:24:05.460158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.62:37129 (local address 127.31.100.3:37559)
0812 05:24:05.460285 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0812 05:24:05.460321 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.100.62:37129: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":61,"threads_started":1}
I20260812 05:24:05.540042 28563 raft_consensus.cc:482] T 4caedac3e22b4e589962ff7140e69aaa P 3f75ed97c1aa4752b59f6a1551419390 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 05:24:05.540139 28563 raft_consensus.cc:504] T 4caedac3e22b4e589962ff7140e69aaa P 3f75ed97c1aa4752b59f6a1551419390 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df8db312fb6c4c57aa51f490c25fa09f" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 45871 } } peers { permanent_uuid: "edf25888a7e648348650523730de9401" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 39191 } } peers { permanent_uuid: "3f75ed97c1aa4752b59f6a1551419390" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 37667 } }
I20260812 05:24:05.540273 28563 leader_election.cc:288] T 4caedac3e22b4e589962ff7140e69aaa P 3f75ed97c1aa4752b59f6a1551419390 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers df8db312fb6c4c57aa51f490c25fa09f (127.31.100.3:45871), edf25888a7e648348650523730de9401 (127.31.100.2:39191)
W20260812 05:24:05.540644 27798 leader_election.cc:341] T 4caedac3e22b4e589962ff7140e69aaa P 3f75ed97c1aa4752b59f6a1551419390 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer df8db312fb6c4c57aa51f490c25fa09f (127.31.100.3:45871): Not found: Tablet not found: 4caedac3e22b4e589962ff7140e69aaa
W20260812 05:24:05.540711 27798 leader_election.cc:341] T 4caedac3e22b4e589962ff7140e69aaa P 3f75ed97c1aa4752b59f6a1551419390 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer edf25888a7e648348650523730de9401 (127.31.100.2:39191): Not found: Tablet not found: 4caedac3e22b4e589962ff7140e69aaa
I20260812 05:24:05.540727 27798 leader_election.cc:302] T 4caedac3e22b4e589962ff7140e69aaa P 3f75ed97c1aa4752b59f6a1551419390 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f75ed97c1aa4752b59f6a1551419390; no voters: df8db312fb6c4c57aa51f490c25fa09f, edf25888a7e648348650523730de9401
I20260812 05:24:05.540795 28563 raft_consensus.cc:2726] T 4caedac3e22b4e589962ff7140e69aaa P 3f75ed97c1aa4752b59f6a1551419390 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260812 05:24:06.264674 28564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:24:06.264460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.62:37129 (local address 127.31.100.1:41825)
0812 05:24:06.264591 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0812 05:24:06.264629 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.100.62:37129: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":81,"threads_started":1}
I20260812 05:24:06.321699 28566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:24:06.321525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.62:37129 (local address 127.31.100.2:47795)
0812 05:24:06.321628 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0812 05:24:06.321660 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.100.62:37129: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89,"thread_start_us":56,"threads_started":1}
I20260812 05:24:06.460947 28567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:24:06.460756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.62:37129 (local address 127.31.100.3:34731)
0812 05:24:06.460869 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0812 05:24:06.460903 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.100.62:37129: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":61,"threads_started":1}
I20260812 05:24:07.265300 28568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:24:07.265087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.62:37129 (local address 127.31.100.1:48355)
0812 05:24:07.265213 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0812 05:24:07.265254 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.100.62:37129: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":65,"threads_started":1}
I20260812 05:24:07.322240 28573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:24:07.322036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.62:37129 (local address 127.31.100.2:37989)
0812 05:24:07.322152 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0812 05:24:07.322201 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.100.62:37129: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":56,"threads_started":1}
I20260812 05:24:07.461591 28579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:24:07.461358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.62:37129 (local address 127.31.100.3:51537)
0812 05:24:07.461508 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0812 05:24:07.461546 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.100.62:37129: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":99,"threads_started":1}
I20260812 05:24:08.265969 28581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:24:08.265733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.62:37129 (local address 127.31.100.1:59075)
0812 05:24:08.265886 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0812 05:24:08.265924 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.100.62:37129: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":83,"threads_started":1}
I20260812 05:24:08.322813 28582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:24:08.322601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.62:37129 (local address 127.31.100.2:44381)
0812 05:24:08.322742 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0812 05:24:08.322774 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.100.62:37129: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":72,"threads_started":1}
I20260812 05:24:08.462203 28583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:24:08.461986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.62:37129 (local address 127.31.100.3:41393)
0812 05:24:08.462127 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0812 05:24:08.462161 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.100.62:37129: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":89,"threads_started":1}
I20260812 05:24:09.266623 28584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:24:09.266381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.62:37129 (local address 127.31.100.1:41687)
0812 05:24:09.266536 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0812 05:24:09.266573 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.100.62:37129: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":100,"threads_started":1}
I20260812 05:24:09.323365 28588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:24:09.323177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.62:37129 (local address 127.31.100.2:46563)
0812 05:24:09.323298 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0812 05:24:09.323326 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.100.62:37129: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":71,"threads_started":1}
I20260812 05:24:09.462769 28590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:24:09.462564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.62:37129 (local address 127.31.100.3:58155)
0812 05:24:09.462693 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0812 05:24:09.462726 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.100.62:37129: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":81,"threads_started":1}
I20260812 05:24:10.267280 28591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:24:10.267062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.62:37129 (local address 127.31.100.1:47449)
0812 05:24:10.267191 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0812 05:24:10.267232 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.100.62:37129: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":64,"threads_started":1}
I20260812 05:24:10.323984 28592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:24:10.323794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.62:37129 (local address 127.31.100.2:41391)
0812 05:24:10.323905 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0812 05:24:10.323941 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.100.62:37129: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":62,"threads_started":1}
I20260812 05:24:10.463397 28593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:24:10.463202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.62:37129 (local address 127.31.100.3:53585)
0812 05:24:10.463316 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0812 05:24:10.463349 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.100.62:37129: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":64,"threads_started":1}
I20260812 05:24:11.172353 28594 raft_consensus.cc:482] T 4caedac3e22b4e589962ff7140e69aaa P 3f75ed97c1aa4752b59f6a1551419390 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 05:24:11.172448 28594 raft_consensus.cc:504] T 4caedac3e22b4e589962ff7140e69aaa P 3f75ed97c1aa4752b59f6a1551419390 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df8db312fb6c4c57aa51f490c25fa09f" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 45871 } } peers { permanent_uuid: "edf25888a7e648348650523730de9401" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 39191 } } peers { permanent_uuid: "3f75ed97c1aa4752b59f6a1551419390" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 37667 } }
I20260812 05:24:11.172618 28594 leader_election.cc:288] T 4caedac3e22b4e589962ff7140e69aaa P 3f75ed97c1aa4752b59f6a1551419390 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers df8db312fb6c4c57aa51f490c25fa09f (127.31.100.3:45871), edf25888a7e648348650523730de9401 (127.31.100.2:39191)
W20260812 05:24:11.173015 27798 leader_election.cc:341] T 4caedac3e22b4e589962ff7140e69aaa P 3f75ed97c1aa4752b59f6a1551419390 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer df8db312fb6c4c57aa51f490c25fa09f (127.31.100.3:45871): Not found: Tablet not found: 4caedac3e22b4e589962ff7140e69aaa
W20260812 05:24:11.173108 27798 leader_election.cc:341] T 4caedac3e22b4e589962ff7140e69aaa P 3f75ed97c1aa4752b59f6a1551419390 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer edf25888a7e648348650523730de9401 (127.31.100.2:39191): Not found: Tablet not found: 4caedac3e22b4e589962ff7140e69aaa
I20260812 05:24:11.173136 27798 leader_election.cc:302] T 4caedac3e22b4e589962ff7140e69aaa P 3f75ed97c1aa4752b59f6a1551419390 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f75ed97c1aa4752b59f6a1551419390; no voters: df8db312fb6c4c57aa51f490c25fa09f, edf25888a7e648348650523730de9401
I20260812 05:24:11.173223 28594 raft_consensus.cc:2726] T 4caedac3e22b4e589962ff7140e69aaa P 3f75ed97c1aa4752b59f6a1551419390 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260812 05:24:11.267903 28595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:24:11.267717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.62:37129 (local address 127.31.100.1:51923)
0812 05:24:11.267827 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0812 05:24:11.267859 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.100.62:37129: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":63,"threads_started":1}
I20260812 05:24:11.324584 28597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:24:11.324391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.62:37129 (local address 127.31.100.2:42387)
0812 05:24:11.324502 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0812 05:24:11.324532 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.100.62:37129: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":61,"threads_started":1}
I20260812 05:24:11.464100 28598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:24:11.463863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.62:37129 (local address 127.31.100.3:47463)
0812 05:24:11.464015 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0812 05:24:11.464055 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.100.62:37129: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":64,"threads_started":1}
I20260812 05:24:11.468905 32144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0waFvT/build/release/bin/kudu with pid 27711
I20260812 05:24:11.482383 32144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0waFvT/build/release/bin/kudu with pid 28009
I20260812 05:24:11.492218 32144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0waFvT/build/release/bin/kudu with pid 28261
2026-08-12T05:24:11Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (58713 ms)
[----------] 30 tests from SecurityITest (233890 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786512017613366-32144-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 05:24:11 dist-test-slave-42z9 krb5kdc[28601](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 05:24:11 dist-test-slave-42z9 krb5kdc[28601](info): set up 2 sockets
Aug 12 05:24:11 dist-test-slave-42z9 krb5kdc[28601](info): commencing operation
krb5kdc: starting...
W20260812 05:24:12.657869 32144 mini_kdc.cc:121] Time spent starting KDC: real 1.142s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786512017613366-32144-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786512017613366-32144-0/krb5kdc/test-user.keytab.
Aug 12 05:24:12 dist-test-slave-42z9 krb5kdc[28601](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512252, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T05:24:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T05:24:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.100.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.100.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.100.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.62.keytab.
Entry for principal kudu/127.31.100.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.62.keytab.
I20260812 05:24:12.771914 32144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0waFvT/build/release/bin/kudu
/tmp/dist-test-task0waFvT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786512017613366-32144-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786512017613366-32144-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786512017613366-32144-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786512017613366-32144-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.100.62:41215
--webserver_interface=127.31.100.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.62.keytab
--principal=kudu/127.31.100.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.100.20:39545
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.100.62:41215
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786512017613366-32144-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786512017613366-32144-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786512017613366-32144-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 05:24:12.851676 28625 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 05:24:12.851877 28625 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 05:24:12.851899 28625 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 05:24:12.853048 28625 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 05:24:12.853092 28625 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 05:24:12.853106 28625 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 05:24:12.853117 28625 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 05:24:12.853140 28625 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 05:24:12.854660 28625 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.100.20:39545
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786512017613366-32144-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786512017613366-32144-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.100.62:41215
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.62.keytab
--principal=kudu/127.31.100.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.100.62:41215
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786512017613366-32144-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.100.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28625
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786512017613366-32144-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 04:20:34 UTC on ccycloud.quasar-phqdqp.root.comops.site
build id 4581871
I20260812 05:24:12.854887 28625 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 05:24:12.855095 28625 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260812 05:24:12.859911 28625 server_base.cc:1034] running on GCE node
W20260812 05:24:12.859951 28639 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 05:24:12.859932 28637 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 05:24:12.860144 28645 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 05:24:12.860376 28625 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 05:24:12.860635 28625 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 05:24:12.861761 28625 hybrid_clock.cc:648] HybridClock initialized: now 1786512252861750 us; error 29 us; skew 500 ppm
Aug 12 05:24:12 dist-test-slave-42z9 krb5kdc[28601](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512252, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.100.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 05:24:12.864037 28625 init.cc:377] Logged in from keytab as kudu/127.31.100.62@KRBTEST.COM (short username kudu)
I20260812 05:24:12.864409 28625 webserver.cc:460] Webserver started at http://127.31.100.62:45529/ using document root <none> and password file <none>
I20260812 05:24:12.864615 28625 fs_manager.cc:359] Metadata directory not provided
I20260812 05:24:12.864663 28625 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 05:24:12.864785 28625 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 05:24:12.865796 28625 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786512017613366-32144-0/minicluster-data/master-0/data/instance:
uuid: "38eaa403a01d42878a061129ceba1203"
format_stamp: "Formatted at 2026-08-12 05:24:12 on dist-test-slave-42z9"
server_key: "c9a7a4617ae79e5444a94a86d6831214"
server_key_iv: "ad1caba38d879079b07551ef121c11cc"
server_key_version: "encryptionkey@0"
I20260812 05:24:12.866115 28625 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786512017613366-32144-0/minicluster-data/master-0/wal/instance:
uuid: "38eaa403a01d42878a061129ceba1203"
format_stamp: "Formatted at 2026-08-12 05:24:12 on dist-test-slave-42z9"
server_key: "c9a7a4617ae79e5444a94a86d6831214"
server_key_iv: "ad1caba38d879079b07551ef121c11cc"
server_key_version: "encryptionkey@0"
I20260812 05:24:12.867296 28625 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260812 05:24:12.867933 28658 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 05:24:12.868075 28625 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260812 05:24:12.868158 28625 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786512017613366-32144-0/minicluster-data/master-0/data,/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786512017613366-32144-0/minicluster-data/master-0/wal
uuid: "38eaa403a01d42878a061129ceba1203"
format_stamp: "Formatted at 2026-08-12 05:24:12 on dist-test-slave-42z9"
server_key: "c9a7a4617ae79e5444a94a86d6831214"
server_key_iv: "ad1caba38d879079b07551ef121c11cc"
server_key_version: "encryptionkey@0"
I20260812 05:24:12.868216 28625 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786512017613366-32144-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786512017613366-32144-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786512017613366-32144-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 05:24:12.878772 28625 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 05:24:12.880738 28625 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 05:24:12.880863 28625 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 05:24:12.884691 28625 rpc_server.cc:307] RPC server started. Bound to: 127.31.100.62:41215
I20260812 05:24:12.884722 28758 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.100.62:41215 every 8 connection(s)
I20260812 05:24:12.885023 28625 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786512017613366-32144-0/minicluster-data/master-0/data/info.pb
I20260812 05:24:12.885881 28759 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 05:24:12.886234 32144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0waFvT/build/release/bin/kudu as pid 28625
I20260812 05:24:12.886335 32144 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786512017613366-32144-0/minicluster-data/master-0/wal/instance
I20260812 05:24:12.886499 32144 external_mini_cluster.cc:1428] Setting key e38d8e4b50cdb47e6e8360acfca9383e
I20260812 05:24:12.888628 28759 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 38eaa403a01d42878a061129ceba1203: Bootstrap starting.
I20260812 05:24:12.889213 28759 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 38eaa403a01d42878a061129ceba1203: Neither blocks nor log segments found. Creating new log.
I20260812 05:24:12.889511 28759 log.cc:824] T 00000000000000000000000000000000 P 38eaa403a01d42878a061129ceba1203: Log is configured to *not* fsync() on all Append() calls
I20260812 05:24:12.890240 28759 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 38eaa403a01d42878a061129ceba1203: No bootstrap required, opened a new log
I20260812 05:24:12.892546 28759 raft_consensus.cc:348] T 00000000000000000000000000000000 P 38eaa403a01d42878a061129ceba1203 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38eaa403a01d42878a061129ceba1203" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 41215 } }
I20260812 05:24:12.892678 28759 raft_consensus.cc:374] T 00000000000000000000000000000000 P 38eaa403a01d42878a061129ceba1203 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 05:24:12.892700 28759 raft_consensus.cc:729] T 00000000000000000000000000000000 P 38eaa403a01d42878a061129ceba1203 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 38eaa403a01d42878a061129ceba1203, State: Initialized, Role: FOLLOWER
I20260812 05:24:12.892807 28759 consensus_queue.cc:260] T 00000000000000000000000000000000 P 38eaa403a01d42878a061129ceba1203 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38eaa403a01d42878a061129ceba1203" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 41215 } }
Aug 12 05:24:12 dist-test-slave-42z9 krb5kdc[28601](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512252, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.100.62@KRBTEST.COM
I20260812 05:24:12.892871 28759 raft_consensus.cc:388] T 00000000000000000000000000000000 P 38eaa403a01d42878a061129ceba1203 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 05:24:12.892894 28759 raft_consensus.cc:482] T 00000000000000000000000000000000 P 38eaa403a01d42878a061129ceba1203 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 05:24:12.892944 28759 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 38eaa403a01d42878a061129ceba1203 [term 0 FOLLOWER]: Advancing to term 1
I20260812 05:24:12.893467 28759 raft_consensus.cc:504] T 00000000000000000000000000000000 P 38eaa403a01d42878a061129ceba1203 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38eaa403a01d42878a061129ceba1203" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 41215 } }
I20260812 05:24:12.893568 28759 leader_election.cc:302] T 00000000000000000000000000000000 P 38eaa403a01d42878a061129ceba1203 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 38eaa403a01d42878a061129ceba1203; no voters:
I20260812 05:24:12.893736 28759 leader_election.cc:288] T 00000000000000000000000000000000 P 38eaa403a01d42878a061129ceba1203 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 05:24:12.893790 28769 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 38eaa403a01d42878a061129ceba1203 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 05:24:12.893993 28759 sys_catalog.cc:564] T 00000000000000000000000000000000 P 38eaa403a01d42878a061129ceba1203 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 05:24:12.894277 28769 raft_consensus.cc:686] T 00000000000000000000000000000000 P 38eaa403a01d42878a061129ceba1203 [term 1 LEADER]: Becoming Leader. State: Replica: 38eaa403a01d42878a061129ceba1203, State: Running, Role: LEADER
I20260812 05:24:12.894426 28769 consensus_queue.cc:237] T 00000000000000000000000000000000 P 38eaa403a01d42878a061129ceba1203 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38eaa403a01d42878a061129ceba1203" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 41215 } }
I20260812 05:24:12.894794 28771 sys_catalog.cc:455] T 00000000000000000000000000000000 P 38eaa403a01d42878a061129ceba1203 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 38eaa403a01d42878a061129ceba1203. Latest consensus state: current_term: 1 leader_uuid: "38eaa403a01d42878a061129ceba1203" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38eaa403a01d42878a061129ceba1203" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 41215 } } }
I20260812 05:24:12.894871 28771 sys_catalog.cc:458] T 00000000000000000000000000000000 P 38eaa403a01d42878a061129ceba1203 [sys.catalog]: This master's current role is: LEADER
I20260812 05:24:12.895103 28770 sys_catalog.cc:455] T 00000000000000000000000000000000 P 38eaa403a01d42878a061129ceba1203 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "38eaa403a01d42878a061129ceba1203" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38eaa403a01d42878a061129ceba1203" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 41215 } } }
I20260812 05:24:12.895224 28770 sys_catalog.cc:458] T 00000000000000000000000000000000 P 38eaa403a01d42878a061129ceba1203 [sys.catalog]: This master's current role is: LEADER
W20260812 05:24:12.895879 28797 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 38eaa403a01d42878a061129ceba1203: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 05:24:12.895967 28797 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 05:24:12.896039 28785 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 05:24:12.896219 28785 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 05:24:12.897055 28764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:24:12.886999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38562 (local address 127.31.100.62:41215)
0812 05:24:12.887147 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0812 05:24:12.887156 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 05:24:12.887180 (+ 24us) server_negotiation.cc:407] Connection header received
0812 05:24:12.887573 (+ 393us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:24:12.887579 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:24:12.887622 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:24:12.887731 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:24:12.888464 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:24:12.889369 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:24:12.890336 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:24:12.890735 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:24:12.890815 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:24:12.893330 (+ 2515us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 05:24:12.893333 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 05:24:12.893334 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 05:24:12.893345 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0812 05:24:12.895235 (+ 1890us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:24:12.895710 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:24:12.895714 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:24:12.895714 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:24:12.895811 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:24:12.896031 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:24:12.896031 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:24:12.896032 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:24:12.896343 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 05:24:12.896456 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0812 05:24:12.896730 (+ 274us) server_negotiation.cc:299] Negotiation successful
0812 05:24:12.896868 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104,"thread_start_us":62,"threads_started":1}
I20260812 05:24:12.897991 28785 catalog_manager.cc:1347] Generated new cluster ID: f387b31bf7964d2da14ee16319ed3356
I20260812 05:24:12.898047 28785 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 05:24:12.904623 28785 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 05:24:12.905104 28785 catalog_manager.cc:1504] Loading token signing keys...
I20260812 05:24:12.911510 28785 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 38eaa403a01d42878a061129ceba1203: Generated new TSK 0
I20260812 05:24:12.911656 28785 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.31.100.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.100.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.100.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.1.keytab.
Entry for principal kudu/127.31.100.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.1.keytab.
I20260812 05:24:12.941696 32144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0waFvT/build/release/bin/kudu
/tmp/dist-test-task0waFvT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786512017613366-32144-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786512017613366-32144-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786512017613366-32144-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786512017613366-32144-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.100.1:0
--local_ip_for_outbound_sockets=127.31.100.1
--webserver_interface=127.31.100.1
--webserver_port=0
--tserver_master_addrs=127.31.100.62:41215
--keytab_file=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.1.keytab
--principal=kudu/127.31.100.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.100.20:39545
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786512017613366-32144-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786512017613366-32144-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786512017613366-32144-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 05:24:13.020815 28809 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 05:24:13.020998 28809 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 05:24:13.021029 28809 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 05:24:13.022145 28809 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 05:24:13.022184 28809 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 05:24:13.022217 28809 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.100.1
I20260812 05:24:13.023620 28809 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.100.20:39545
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786512017613366-32144-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786512017613366-32144-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.1.keytab
--principal=kudu/127.31.100.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.100.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786512017613366-32144-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.100.1
--webserver_port=0
--tserver_master_addrs=127.31.100.62:41215
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28809
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.100.1
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786512017613366-32144-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 04:20:34 UTC on ccycloud.quasar-phqdqp.root.comops.site
build id 4581871
I20260812 05:24:13.023849 28809 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 05:24:13.024060 28809 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 05:24:13.028811 28818 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 05:24:13.028816 28822 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 05:24:13.028838 28819 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 05:24:13.029068 28809 server_base.cc:1034] running on GCE node
I20260812 05:24:13.029268 28809 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 05:24:13.029582 28809 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 05:24:13.030722 28809 hybrid_clock.cc:648] HybridClock initialized: now 1786512253030708 us; error 30 us; skew 500 ppm
Aug 12 05:24:13 dist-test-slave-42z9 krb5kdc[28601](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512253, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.100.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 05:24:13.033089 28809 init.cc:377] Logged in from keytab as kudu/127.31.100.1@KRBTEST.COM (short username kudu)
I20260812 05:24:13.033550 28809 webserver.cc:460] Webserver started at http://127.31.100.1:38755/ using document root <none> and password file <none>
I20260812 05:24:13.033752 28809 fs_manager.cc:359] Metadata directory not provided
I20260812 05:24:13.033790 28809 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 05:24:13.033881 28809 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 05:24:13.034900 28809 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786512017613366-32144-0/minicluster-data/ts-0/data/instance:
uuid: "83e83f183de7453dafefae00cd3560fa"
format_stamp: "Formatted at 2026-08-12 05:24:13 on dist-test-slave-42z9"
server_key: "35ac0cc90207559c1803453994b68740"
server_key_iv: "cfa0c50bb74313fb2c53e6ef1f629c44"
server_key_version: "encryptionkey@0"
I20260812 05:24:13.035195 28809 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786512017613366-32144-0/minicluster-data/ts-0/wal/instance:
uuid: "83e83f183de7453dafefae00cd3560fa"
format_stamp: "Formatted at 2026-08-12 05:24:13 on dist-test-slave-42z9"
server_key: "35ac0cc90207559c1803453994b68740"
server_key_iv: "cfa0c50bb74313fb2c53e6ef1f629c44"
server_key_version: "encryptionkey@0"
I20260812 05:24:13.036365 28809 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260812 05:24:13.037027 28829 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 05:24:13.037190 28809 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260812 05:24:13.037281 28809 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786512017613366-32144-0/minicluster-data/ts-0/data,/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786512017613366-32144-0/minicluster-data/ts-0/wal
uuid: "83e83f183de7453dafefae00cd3560fa"
format_stamp: "Formatted at 2026-08-12 05:24:13 on dist-test-slave-42z9"
server_key: "35ac0cc90207559c1803453994b68740"
server_key_iv: "cfa0c50bb74313fb2c53e6ef1f629c44"
server_key_version: "encryptionkey@0"
I20260812 05:24:13.037348 28809 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786512017613366-32144-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786512017613366-32144-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786512017613366-32144-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 05:24:13.048089 28809 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 05:24:13.049800 28809 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 05:24:13.049923 28809 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 05:24:13.050124 28809 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 05:24:13.050422 28809 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 05:24:13.050457 28809 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 05:24:13.050477 28809 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 05:24:13.050487 28809 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 05:24:13.055713 28809 rpc_server.cc:307] RPC server started. Bound to: 127.31.100.1:39183
I20260812 05:24:13.055781 29027 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.100.1:39183 every 8 connection(s)
I20260812 05:24:13.056042 28809 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786512017613366-32144-0/minicluster-data/ts-0/data/info.pb
I20260812 05:24:13.056401 32144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0waFvT/build/release/bin/kudu as pid 28809
I20260812 05:24:13.056512 32144 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786512017613366-32144-0/minicluster-data/ts-0/wal/instance
I20260812 05:24:13.056770 32144 external_mini_cluster.cc:1428] Setting key 1f8626e3282d7fb632296f13be9cad6a
Aug 12 05:24:13 dist-test-slave-42z9 krb5kdc[28601](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512253, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.100.1@KRBTEST.COM for kudu/127.31.100.62@KRBTEST.COM
I20260812 05:24:13.066210 28764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:24:13.056832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.100.1:44137 (local address 127.31.100.62:41215)
0812 05:24:13.056913 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0812 05:24:13.056915 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 05:24:13.057105 (+ 190us) server_negotiation.cc:407] Connection header received
0812 05:24:13.057756 (+ 651us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:24:13.057757 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:24:13.057769 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:24:13.057810 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:24:13.058967 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:24:13.059475 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:24:13.060831 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:24:13.061116 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:24:13.061188 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:24:13.063365 (+ 2177us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 05:24:13.063366 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 05:24:13.063367 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 05:24:13.063373 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0812 05:24:13.064707 (+ 1334us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:24:13.065181 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:24:13.065182 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:24:13.065183 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:24:13.065236 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:24:13.065519 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:24:13.065520 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:24:13.065520 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:24:13.065643 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 05:24:13.065705 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0812 05:24:13.066047 (+ 342us) server_negotiation.cc:299] Negotiation successful
0812 05:24:13.066140 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260812 05:24:13.066408 29035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:24:13.056727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.62:41215 (local address 127.31.100.1:44137)
0812 05:24:13.056980 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0812 05:24:13.057022 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0812 05:24:13.057663 (+ 641us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:24:13.057877 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:24:13.057881 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:24:13.058594 (+ 713us) client_negotiation.cc:189] Negotiated authn=SASL
0812 05:24:13.058846 (+ 252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:24:13.058852 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:24:13.059557 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:24:13.059558 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:24:13.060734 (+ 1176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:24:13.060738 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:24:13.061191 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:24:13.061192 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:24:13.061299 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:24:13.061759 (+ 460us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 05:24:13.061767 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0812 05:24:13.063252 (+ 1485us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 05:24:13.064792 (+ 1540us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 05:24:13.064800 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 05:24:13.064802 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 05:24:13.065111 (+ 309us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 05:24:13.065297 (+ 186us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 05:24:13.065298 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 05:24:13.065298 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 05:24:13.065432 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 05:24:13.065713 (+ 281us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 05:24:13.065721 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 05:24:13.065890 (+ 169us) client_negotiation.cc:769] Sending connection context
0812 05:24:13.066006 (+ 116us) client_negotiation.cc:240] Negotiation successful
0812 05:24:13.066181 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":89,"threads_started":1}
WARNING: no policy specified for kudu/127.31.100.2@KRBTEST.COM; defaulting to no policy
I20260812 05:24:13.067001 29032 heartbeater.cc:347] Connected to a master server at 127.31.100.62:41215
I20260812 05:24:13.067090 29032 heartbeater.cc:464] Registering TS with master...
I20260812 05:24:13.067233 29032 heartbeater.cc:511] Master 127.31.100.62:41215 requested a full tablet report, sending...
I20260812 05:24:13.067698 28690 ts_manager.cc:194] Registered new tserver with Master: 83e83f183de7453dafefae00cd3560fa (127.31.100.1:39183)
I20260812 05:24:13.068351 28690 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.100.1@KRBTEST.COM'} at 127.31.100.1:44137
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.100.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.100.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.2.keytab.
Entry for principal kudu/127.31.100.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.2.keytab.
I20260812 05:24:13.082163 32144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0waFvT/build/release/bin/kudu
/tmp/dist-test-task0waFvT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786512017613366-32144-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786512017613366-32144-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786512017613366-32144-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786512017613366-32144-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.100.2:0
--local_ip_for_outbound_sockets=127.31.100.2
--webserver_interface=127.31.100.2
--webserver_port=0
--tserver_master_addrs=127.31.100.62:41215
--keytab_file=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.2.keytab
--principal=kudu/127.31.100.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.100.20:39545
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786512017613366-32144-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786512017613366-32144-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786512017613366-32144-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 05:24:13.164054 29038 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 05:24:13.164249 29038 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 05:24:13.164280 29038 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 05:24:13.165390 29038 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 05:24:13.165431 29038 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 05:24:13.165489 29038 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.100.2
I20260812 05:24:13.166996 29038 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.100.20:39545
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786512017613366-32144-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786512017613366-32144-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.2.keytab
--principal=kudu/127.31.100.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.100.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786512017613366-32144-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.100.2
--webserver_port=0
--tserver_master_addrs=127.31.100.62:41215
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29038
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.100.2
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786512017613366-32144-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 04:20:34 UTC on ccycloud.quasar-phqdqp.root.comops.site
build id 4581871
I20260812 05:24:13.167232 29038 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 05:24:13.167430 29038 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 05:24:13.172243 29052 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 05:24:13.172233 29046 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 05:24:13.172233 29047 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 05:24:13.172469 29038 server_base.cc:1034] running on GCE node
I20260812 05:24:13.172673 29038 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 05:24:13.172904 29038 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 05:24:13.174048 29038 hybrid_clock.cc:648] HybridClock initialized: now 1786512253174023 us; error 38 us; skew 500 ppm
Aug 12 05:24:13 dist-test-slave-42z9 krb5kdc[28601](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512253, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.100.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 05:24:13.176332 29038 init.cc:377] Logged in from keytab as kudu/127.31.100.2@KRBTEST.COM (short username kudu)
I20260812 05:24:13.176771 29038 webserver.cc:460] Webserver started at http://127.31.100.2:36823/ using document root <none> and password file <none>
I20260812 05:24:13.176980 29038 fs_manager.cc:359] Metadata directory not provided
I20260812 05:24:13.177027 29038 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 05:24:13.177150 29038 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 05:24:13.178164 29038 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786512017613366-32144-0/minicluster-data/ts-1/data/instance:
uuid: "c115200e338540ec9d442a2d072cf6cb"
format_stamp: "Formatted at 2026-08-12 05:24:13 on dist-test-slave-42z9"
server_key: "26c53998a59507ef2cbf9af5866ec23c"
server_key_iv: "813649ffba52ace4dd9cd3e466d7e01d"
server_key_version: "encryptionkey@0"
I20260812 05:24:13.178486 29038 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786512017613366-32144-0/minicluster-data/ts-1/wal/instance:
uuid: "c115200e338540ec9d442a2d072cf6cb"
format_stamp: "Formatted at 2026-08-12 05:24:13 on dist-test-slave-42z9"
server_key: "26c53998a59507ef2cbf9af5866ec23c"
server_key_iv: "813649ffba52ace4dd9cd3e466d7e01d"
server_key_version: "encryptionkey@0"
I20260812 05:24:13.179687 29038 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260812 05:24:13.180318 29064 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 05:24:13.180464 29038 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260812 05:24:13.180552 29038 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786512017613366-32144-0/minicluster-data/ts-1/data,/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786512017613366-32144-0/minicluster-data/ts-1/wal
uuid: "c115200e338540ec9d442a2d072cf6cb"
format_stamp: "Formatted at 2026-08-12 05:24:13 on dist-test-slave-42z9"
server_key: "26c53998a59507ef2cbf9af5866ec23c"
server_key_iv: "813649ffba52ace4dd9cd3e466d7e01d"
server_key_version: "encryptionkey@0"
I20260812 05:24:13.180608 29038 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786512017613366-32144-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786512017613366-32144-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786512017613366-32144-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 05:24:13.199306 29038 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 05:24:13.200907 29038 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 05:24:13.201032 29038 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 05:24:13.201228 29038 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 05:24:13.201596 29038 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 05:24:13.201635 29038 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 05:24:13.201663 29038 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 05:24:13.201680 29038 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 05:24:13.206920 29038 rpc_server.cc:307] RPC server started. Bound to: 127.31.100.2:46537
I20260812 05:24:13.206975 29269 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.100.2:46537 every 8 connection(s)
I20260812 05:24:13.207243 29038 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786512017613366-32144-0/minicluster-data/ts-1/data/info.pb
Aug 12 05:24:13 dist-test-slave-42z9 krb5kdc[28601](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512253, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.100.2@KRBTEST.COM for kudu/127.31.100.62@KRBTEST.COM
I20260812 05:24:13.215061 28764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:24:13.207867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.100.2:42727 (local address 127.31.100.62:41215)
0812 05:24:13.207929 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0812 05:24:13.207931 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 05:24:13.208075 (+ 144us) server_negotiation.cc:407] Connection header received
0812 05:24:13.208504 (+ 429us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:24:13.208505 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:24:13.208515 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:24:13.208565 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:24:13.209373 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:24:13.209918 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:24:13.210809 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:24:13.210988 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:24:13.211030 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:24:13.212816 (+ 1786us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 05:24:13.212817 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 05:24:13.212818 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 05:24:13.212824 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0812 05:24:13.214077 (+ 1253us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:24:13.214379 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:24:13.214380 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:24:13.214380 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:24:13.214427 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:24:13.214619 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:24:13.214619 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:24:13.214619 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:24:13.214711 (+ 92us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 05:24:13.214755 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0812 05:24:13.214936 (+ 181us) server_negotiation.cc:299] Negotiation successful
0812 05:24:13.214995 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260812 05:24:13.215200 29275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:24:13.207822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.62:41215 (local address 127.31.100.2:42727)
0812 05:24:13.207962 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0812 05:24:13.208001 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0812 05:24:13.208434 (+ 433us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:24:13.208615 (+ 181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:24:13.208618 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:24:13.209133 (+ 515us) client_negotiation.cc:189] Negotiated authn=SASL
0812 05:24:13.209311 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:24:13.209315 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:24:13.209986 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:24:13.209986 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:24:13.210748 (+ 762us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:24:13.210750 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:24:13.211038 (+ 288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:24:13.211038 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:24:13.211134 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:24:13.211505 (+ 371us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 05:24:13.211513 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0812 05:24:13.212733 (+ 1220us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 05:24:13.214135 (+ 1402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 05:24:13.214139 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 05:24:13.214140 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 05:24:13.214326 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 05:24:13.214470 (+ 144us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 05:24:13.214470 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 05:24:13.214471 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 05:24:13.214569 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 05:24:13.214751 (+ 182us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 05:24:13.214753 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 05:24:13.214849 (+ 96us) client_negotiation.cc:769] Sending connection context
0812 05:24:13.214919 (+ 70us) client_negotiation.cc:240] Negotiation successful
0812 05:24:13.215026 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":75,"threads_started":1}
I20260812 05:24:13.215646 29271 heartbeater.cc:347] Connected to a master server at 127.31.100.62:41215
I20260812 05:24:13.215728 29271 heartbeater.cc:464] Registering TS with master...
I20260812 05:24:13.215883 29271 heartbeater.cc:511] Master 127.31.100.62:41215 requested a full tablet report, sending...
I20260812 05:24:13.216207 28690 ts_manager.cc:194] Registered new tserver with Master: c115200e338540ec9d442a2d072cf6cb (127.31.100.2:46537)
I20260812 05:24:13.216537 28690 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.100.2@KRBTEST.COM'} at 127.31.100.2:42727
I20260812 05:24:13.216771 32144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0waFvT/build/release/bin/kudu as pid 29038
I20260812 05:24:13.216863 32144 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786512017613366-32144-0/minicluster-data/ts-1/wal/instance
I20260812 05:24:13.217039 32144 external_mini_cluster.cc:1428] Setting key 0cef13b28fbf2dc50695b0dfac44e816
WARNING: no policy specified for kudu/127.31.100.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.100.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.100.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.3.keytab.
Entry for principal kudu/127.31.100.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.3.keytab.
I20260812 05:24:13.239952 32144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0waFvT/build/release/bin/kudu
/tmp/dist-test-task0waFvT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786512017613366-32144-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786512017613366-32144-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786512017613366-32144-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786512017613366-32144-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.100.3:0
--local_ip_for_outbound_sockets=127.31.100.3
--webserver_interface=127.31.100.3
--webserver_port=0
--tserver_master_addrs=127.31.100.62:41215
--keytab_file=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.3.keytab
--principal=kudu/127.31.100.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.100.20:39545
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786512017613366-32144-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786512017613366-32144-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786512017613366-32144-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 05:24:13.319101 29279 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 05:24:13.319274 29279 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 05:24:13.319295 29279 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 05:24:13.320399 29279 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 05:24:13.320438 29279 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 05:24:13.320466 29279 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.100.3
I20260812 05:24:13.321904 29279 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.100.20:39545
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786512017613366-32144-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786512017613366-32144-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.3.keytab
--principal=kudu/127.31.100.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.100.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786512017613366-32144-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.100.3
--webserver_port=0
--tserver_master_addrs=127.31.100.62:41215
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29279
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.100.3
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786512017613366-32144-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 04:20:34 UTC on ccycloud.quasar-phqdqp.root.comops.site
build id 4581871
I20260812 05:24:13.322109 29279 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 05:24:13.322309 29279 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 05:24:13.326859 29292 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 05:24:13.326967 29279 server_base.cc:1034] running on GCE node
W20260812 05:24:13.326889 29288 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 05:24:13.326920 29286 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 05:24:13.327241 29279 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 05:24:13.327476 29279 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 05:24:13.328617 29279 hybrid_clock.cc:648] HybridClock initialized: now 1786512253328601 us; error 29 us; skew 500 ppm
Aug 12 05:24:13 dist-test-slave-42z9 krb5kdc[28601](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512253, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.100.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 05:24:13.330849 29279 init.cc:377] Logged in from keytab as kudu/127.31.100.3@KRBTEST.COM (short username kudu)
I20260812 05:24:13.331257 29279 webserver.cc:460] Webserver started at http://127.31.100.3:37965/ using document root <none> and password file <none>
I20260812 05:24:13.331461 29279 fs_manager.cc:359] Metadata directory not provided
I20260812 05:24:13.331507 29279 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 05:24:13.331630 29279 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 05:24:13.332592 29279 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786512017613366-32144-0/minicluster-data/ts-2/data/instance:
uuid: "c23f1ed3d1e747dbbcdcd9152af7718f"
format_stamp: "Formatted at 2026-08-12 05:24:13 on dist-test-slave-42z9"
server_key: "1944f5bebae1c21ac16b317165d7952a"
server_key_iv: "7c114ad26d6ad07b13aaa42d5b524491"
server_key_version: "encryptionkey@0"
I20260812 05:24:13.332904 29279 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786512017613366-32144-0/minicluster-data/ts-2/wal/instance:
uuid: "c23f1ed3d1e747dbbcdcd9152af7718f"
format_stamp: "Formatted at 2026-08-12 05:24:13 on dist-test-slave-42z9"
server_key: "1944f5bebae1c21ac16b317165d7952a"
server_key_iv: "7c114ad26d6ad07b13aaa42d5b524491"
server_key_version: "encryptionkey@0"
I20260812 05:24:13.334188 29279 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260812 05:24:13.334935 29302 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 05:24:13.335116 29279 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260812 05:24:13.335207 29279 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786512017613366-32144-0/minicluster-data/ts-2/data,/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786512017613366-32144-0/minicluster-data/ts-2/wal
uuid: "c23f1ed3d1e747dbbcdcd9152af7718f"
format_stamp: "Formatted at 2026-08-12 05:24:13 on dist-test-slave-42z9"
server_key: "1944f5bebae1c21ac16b317165d7952a"
server_key_iv: "7c114ad26d6ad07b13aaa42d5b524491"
server_key_version: "encryptionkey@0"
I20260812 05:24:13.335266 29279 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786512017613366-32144-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786512017613366-32144-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786512017613366-32144-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 05:24:13.368242 29279 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 05:24:13.370103 29279 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 05:24:13.370239 29279 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 05:24:13.370456 29279 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 05:24:13.370745 29279 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 05:24:13.370779 29279 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 05:24:13.370805 29279 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 05:24:13.370829 29279 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 05:24:13.376408 29279 rpc_server.cc:307] RPC server started. Bound to: 127.31.100.3:35489
I20260812 05:24:13.376447 29476 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.100.3:35489 every 8 connection(s)
I20260812 05:24:13.376731 29279 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786512017613366-32144-0/minicluster-data/ts-2/data/info.pb
Aug 12 05:24:13 dist-test-slave-42z9 krb5kdc[28601](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512253, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.100.3@KRBTEST.COM for kudu/127.31.100.62@KRBTEST.COM
I20260812 05:24:13.384634 32144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0waFvT/build/release/bin/kudu as pid 29279
I20260812 05:24:13.384749 32144 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786512017613366-32144-0/minicluster-data/ts-2/wal/instance
I20260812 05:24:13.384928 32144 external_mini_cluster.cc:1428] Setting key 336edf9490cbe830eb411b5b4ffdbf00
I20260812 05:24:13.385403 28764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:24:13.377370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.100.3:60007 (local address 127.31.100.62:41215)
0812 05:24:13.377427 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0812 05:24:13.377429 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 05:24:13.377617 (+ 188us) server_negotiation.cc:407] Connection header received
0812 05:24:13.378126 (+ 509us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:24:13.378127 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:24:13.378141 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:24:13.378192 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:24:13.379080 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:24:13.379598 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:24:13.380537 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:24:13.380749 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:24:13.380805 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:24:13.382812 (+ 2007us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 05:24:13.382814 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 05:24:13.382814 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 05:24:13.382822 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0812 05:24:13.384076 (+ 1254us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:24:13.384467 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:24:13.384468 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:24:13.384468 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:24:13.384522 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:24:13.384771 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:24:13.384772 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:24:13.384772 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:24:13.384894 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 05:24:13.384947 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0812 05:24:13.385226 (+ 279us) server_negotiation.cc:299] Negotiation successful
0812 05:24:13.385311 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260812 05:24:13.385535 29481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:24:13.377338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.62:41215 (local address 127.31.100.3:60007)
0812 05:24:13.377507 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0812 05:24:13.377541 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0812 05:24:13.378022 (+ 481us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:24:13.378258 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:24:13.378262 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:24:13.378803 (+ 541us) client_negotiation.cc:189] Negotiated authn=SASL
0812 05:24:13.378986 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:24:13.378991 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:24:13.379685 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:24:13.379686 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:24:13.380454 (+ 768us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:24:13.380456 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:24:13.380818 (+ 362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:24:13.380819 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:24:13.380929 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:24:13.381349 (+ 420us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 05:24:13.381357 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0812 05:24:13.382728 (+ 1371us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 05:24:13.384187 (+ 1459us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 05:24:13.384192 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 05:24:13.384194 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 05:24:13.384402 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 05:24:13.384573 (+ 171us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 05:24:13.384574 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 05:24:13.384575 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 05:24:13.384711 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 05:24:13.384959 (+ 248us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 05:24:13.384966 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 05:24:13.385111 (+ 145us) client_negotiation.cc:769] Sending connection context
0812 05:24:13.385195 (+ 84us) client_negotiation.cc:240] Negotiation successful
0812 05:24:13.385312 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":74,"threads_started":1}
I20260812 05:24:13.385947 29478 heartbeater.cc:347] Connected to a master server at 127.31.100.62:41215
I20260812 05:24:13.386034 29478 heartbeater.cc:464] Registering TS with master...
I20260812 05:24:13.386204 29478 heartbeater.cc:511] Master 127.31.100.62:41215 requested a full tablet report, sending...
I20260812 05:24:13.386530 28690 ts_manager.cc:194] Registered new tserver with Master: c23f1ed3d1e747dbbcdcd9152af7718f (127.31.100.3:35489)
I20260812 05:24:13.386858 28690 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.100.3@KRBTEST.COM'} at 127.31.100.3:60007
I20260812 05:24:13.397346 32144 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 05:24:13.403476 28764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:24:13.398561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38572 (local address 127.31.100.62:41215)
0812 05:24:13.398596 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0812 05:24:13.398597 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0812 05:24:13.398690 (+ 93us) server_negotiation.cc:407] Connection header received
0812 05:24:13.398728 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:24:13.398729 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:24:13.398737 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:24:13.398778 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:24:13.399228 (+ 450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:24:13.399693 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:24:13.400323 (+ 630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:24:13.400492 (+ 169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:24:13.400536 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:24:13.401484 (+ 948us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 05:24:13.401485 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 05:24:13.401486 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 05:24:13.401492 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0812 05:24:13.402682 (+ 1190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:24:13.402937 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:24:13.402938 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:24:13.402938 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:24:13.402982 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:24:13.403091 (+ 109us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:24:13.403092 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:24:13.403092 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:24:13.403217 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 05:24:13.403286 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0812 05:24:13.403359 (+ 73us) server_negotiation.cc:299] Negotiation successful
0812 05:24:13.403417 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260812 05:24:13.404765 28690 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38572:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 05:24:13.406383 28690 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 05:24:13.412237 29494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:24:13.408336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.3:35489 (local address 127.0.0.1:40154)
0812 05:24:13.408706 (+ 370us) negotiation.cc:107] Waiting for socket to connect
0812 05:24:13.408744 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0812 05:24:13.408809 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:24:13.409006 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:24:13.409010 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:24:13.409012 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 05:24:13.409074 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:24:13.409076 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:24:13.410362 (+ 1286us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:24:13.410363 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:24:13.411213 (+ 850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:24:13.411216 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:24:13.411952 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:24:13.411953 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:24:13.412028 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:24:13.412042 (+ 14us) client_negotiation.cc:769] Sending connection context
0812 05:24:13.412104 (+ 62us) client_negotiation.cc:240] Negotiation successful
0812 05:24:13.412154 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":347,"mutex_wait_us":34,"thread_start_us":59,"threads_started":1}
I20260812 05:24:13.412627 29503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:24:13.408558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40154 (local address 127.31.100.3:35489)
0812 05:24:13.408710 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0812 05:24:13.408715 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 05:24:13.408807 (+ 92us) server_negotiation.cc:407] Connection header received
0812 05:24:13.408887 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:24:13.408889 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:24:13.408904 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:24:13.408984 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 05:24:13.409155 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:24:13.410255 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:24:13.411297 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:24:13.411882 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:24:13.411953 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:24:13.412004 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0812 05:24:13.412510 (+ 506us) server_negotiation.cc:299] Negotiation successful
0812 05:24:13.412554 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":75,"threads_started":1}
I20260812 05:24:13.412662 29499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:24:13.408525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45658 (local address 127.31.100.1:39183)
0812 05:24:13.408686 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0812 05:24:13.408691 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 05:24:13.408963 (+ 272us) server_negotiation.cc:407] Connection header received
0812 05:24:13.409032 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:24:13.409035 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:24:13.409051 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:24:13.409123 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 05:24:13.409396 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:24:13.410116 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:24:13.411585 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:24:13.412326 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:24:13.412393 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:24:13.412449 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0812 05:24:13.412539 (+ 90us) server_negotiation.cc:299] Negotiation successful
0812 05:24:13.412584 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":79,"threads_started":1}
I20260812 05:24:13.412626 29495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:24:13.408336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.1:39183 (local address 127.0.0.1:45658)
0812 05:24:13.408907 (+ 571us) negotiation.cc:107] Waiting for socket to connect
0812 05:24:13.408916 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0812 05:24:13.408961 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:24:13.409193 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:24:13.409194 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:24:13.409195 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 05:24:13.409243 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:24:13.409244 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:24:13.410202 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:24:13.410203 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:24:13.411498 (+ 1295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:24:13.411501 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:24:13.412401 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:24:13.412402 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:24:13.412466 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:24:13.412470 (+ 4us) client_negotiation.cc:769] Sending connection context
0812 05:24:13.412522 (+ 52us) client_negotiation.cc:240] Negotiation successful
0812 05:24:13.412560 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":562,"mutex_wait_us":30,"thread_start_us":62,"threads_started":1}
I20260812 05:24:13.412858 29496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:24:13.408341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.2:46537 (local address 127.0.0.1:60140)
0812 05:24:13.409025 (+ 684us) negotiation.cc:107] Waiting for socket to connect
0812 05:24:13.409032 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0812 05:24:13.409080 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:24:13.409243 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:24:13.409244 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:24:13.409245 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 05:24:13.409278 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:24:13.409280 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:24:13.410558 (+ 1278us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:24:13.410560 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:24:13.411628 (+ 1068us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:24:13.411630 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:24:13.412641 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:24:13.412642 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:24:13.412709 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:24:13.412714 (+ 5us) client_negotiation.cc:769] Sending connection context
0812 05:24:13.412764 (+ 50us) client_negotiation.cc:240] Negotiation successful
0812 05:24:13.412799 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":676,"thread_start_us":51,"threads_started":1}
I20260812 05:24:13.413085 29504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:24:13.408740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60140 (local address 127.31.100.2:46537)
0812 05:24:13.408909 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0812 05:24:13.408913 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 05:24:13.409082 (+ 169us) server_negotiation.cc:407] Connection header received
0812 05:24:13.409142 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:24:13.409145 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:24:13.409161 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:24:13.409240 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 05:24:13.409406 (+ 166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:24:13.410464 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:24:13.411696 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:24:13.412490 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:24:13.412553 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:24:13.412606 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0812 05:24:13.412967 (+ 361us) server_negotiation.cc:299] Negotiation successful
0812 05:24:13.413015 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":74,"threads_started":1}
I20260812 05:24:13.413511 29378 tablet_service.cc:1467] Processing CreateTablet for tablet a2efd5e529db4df084288179f737a287 (DEFAULT_TABLE table=test-table [id=bfd02d1322a7417598cee7010251de76]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 05:24:13.413554 28923 tablet_service.cc:1467] Processing CreateTablet for tablet a2efd5e529db4df084288179f737a287 (DEFAULT_TABLE table=test-table [id=bfd02d1322a7417598cee7010251de76]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 05:24:13.413702 29157 tablet_service.cc:1467] Processing CreateTablet for tablet a2efd5e529db4df084288179f737a287 (DEFAULT_TABLE table=test-table [id=bfd02d1322a7417598cee7010251de76]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 05:24:13.413847 29378 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a2efd5e529db4df084288179f737a287. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 05:24:13.413902 28923 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a2efd5e529db4df084288179f737a287. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 05:24:13.413966 29157 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a2efd5e529db4df084288179f737a287. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 05:24:13.416275 29505 tablet_bootstrap.cc:492] T a2efd5e529db4df084288179f737a287 P c23f1ed3d1e747dbbcdcd9152af7718f: Bootstrap starting.
I20260812 05:24:13.416910 29505 tablet_bootstrap.cc:654] T a2efd5e529db4df084288179f737a287 P c23f1ed3d1e747dbbcdcd9152af7718f: Neither blocks nor log segments found. Creating new log.
I20260812 05:24:13.417114 29507 tablet_bootstrap.cc:492] T a2efd5e529db4df084288179f737a287 P 83e83f183de7453dafefae00cd3560fa: Bootstrap starting.
I20260812 05:24:13.417557 29505 log.cc:824] T a2efd5e529db4df084288179f737a287 P c23f1ed3d1e747dbbcdcd9152af7718f: Log is configured to *not* fsync() on all Append() calls
I20260812 05:24:13.417586 29508 tablet_bootstrap.cc:492] T a2efd5e529db4df084288179f737a287 P c115200e338540ec9d442a2d072cf6cb: Bootstrap starting.
I20260812 05:24:13.417832 29507 tablet_bootstrap.cc:654] T a2efd5e529db4df084288179f737a287 P 83e83f183de7453dafefae00cd3560fa: Neither blocks nor log segments found. Creating new log.
I20260812 05:24:13.418170 29508 tablet_bootstrap.cc:654] T a2efd5e529db4df084288179f737a287 P c115200e338540ec9d442a2d072cf6cb: Neither blocks nor log segments found. Creating new log.
I20260812 05:24:13.418346 29505 tablet_bootstrap.cc:492] T a2efd5e529db4df084288179f737a287 P c23f1ed3d1e747dbbcdcd9152af7718f: No bootstrap required, opened a new log
I20260812 05:24:13.418355 29507 log.cc:824] T a2efd5e529db4df084288179f737a287 P 83e83f183de7453dafefae00cd3560fa: Log is configured to *not* fsync() on all Append() calls
I20260812 05:24:13.418421 29505 ts_tablet_manager.cc:1397] T a2efd5e529db4df084288179f737a287 P c23f1ed3d1e747dbbcdcd9152af7718f: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260812 05:24:13.418694 29508 log.cc:824] T a2efd5e529db4df084288179f737a287 P c115200e338540ec9d442a2d072cf6cb: Log is configured to *not* fsync() on all Append() calls
I20260812 05:24:13.419122 29507 tablet_bootstrap.cc:492] T a2efd5e529db4df084288179f737a287 P 83e83f183de7453dafefae00cd3560fa: No bootstrap required, opened a new log
I20260812 05:24:13.419287 29507 ts_tablet_manager.cc:1397] T a2efd5e529db4df084288179f737a287 P 83e83f183de7453dafefae00cd3560fa: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260812 05:24:13.419413 29508 tablet_bootstrap.cc:492] T a2efd5e529db4df084288179f737a287 P c115200e338540ec9d442a2d072cf6cb: No bootstrap required, opened a new log
I20260812 05:24:13.419476 29508 ts_tablet_manager.cc:1397] T a2efd5e529db4df084288179f737a287 P c115200e338540ec9d442a2d072cf6cb: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260812 05:24:13.420228 29505 raft_consensus.cc:348] T a2efd5e529db4df084288179f737a287 P c23f1ed3d1e747dbbcdcd9152af7718f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e83f183de7453dafefae00cd3560fa" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 39183 } } peers { permanent_uuid: "c115200e338540ec9d442a2d072cf6cb" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 46537 } } peers { permanent_uuid: "c23f1ed3d1e747dbbcdcd9152af7718f" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 35489 } }
I20260812 05:24:13.420400 29505 raft_consensus.cc:374] T a2efd5e529db4df084288179f737a287 P c23f1ed3d1e747dbbcdcd9152af7718f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 05:24:13.420436 29505 raft_consensus.cc:729] T a2efd5e529db4df084288179f737a287 P c23f1ed3d1e747dbbcdcd9152af7718f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c23f1ed3d1e747dbbcdcd9152af7718f, State: Initialized, Role: FOLLOWER
I20260812 05:24:13.420581 29505 consensus_queue.cc:260] T a2efd5e529db4df084288179f737a287 P c23f1ed3d1e747dbbcdcd9152af7718f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e83f183de7453dafefae00cd3560fa" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 39183 } } peers { permanent_uuid: "c115200e338540ec9d442a2d072cf6cb" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 46537 } } peers { permanent_uuid: "c23f1ed3d1e747dbbcdcd9152af7718f" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 35489 } }
I20260812 05:24:13.420874 29505 ts_tablet_manager.cc:1428] T a2efd5e529db4df084288179f737a287 P c23f1ed3d1e747dbbcdcd9152af7718f: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260812 05:24:13.421015 29478 heartbeater.cc:503] Master 127.31.100.62:41215 was elected leader, sending a full tablet report...
I20260812 05:24:13.421147 29507 raft_consensus.cc:348] T a2efd5e529db4df084288179f737a287 P 83e83f183de7453dafefae00cd3560fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e83f183de7453dafefae00cd3560fa" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 39183 } } peers { permanent_uuid: "c115200e338540ec9d442a2d072cf6cb" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 46537 } } peers { permanent_uuid: "c23f1ed3d1e747dbbcdcd9152af7718f" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 35489 } }
I20260812 05:24:13.421265 29507 raft_consensus.cc:374] T a2efd5e529db4df084288179f737a287 P 83e83f183de7453dafefae00cd3560fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 05:24:13.421187 29508 raft_consensus.cc:348] T a2efd5e529db4df084288179f737a287 P c115200e338540ec9d442a2d072cf6cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e83f183de7453dafefae00cd3560fa" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 39183 } } peers { permanent_uuid: "c115200e338540ec9d442a2d072cf6cb" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 46537 } } peers { permanent_uuid: "c23f1ed3d1e747dbbcdcd9152af7718f" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 35489 } }
I20260812 05:24:13.421342 29507 raft_consensus.cc:729] T a2efd5e529db4df084288179f737a287 P 83e83f183de7453dafefae00cd3560fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83e83f183de7453dafefae00cd3560fa, State: Initialized, Role: FOLLOWER
I20260812 05:24:13.421352 29508 raft_consensus.cc:374] T a2efd5e529db4df084288179f737a287 P c115200e338540ec9d442a2d072cf6cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 05:24:13.421396 29508 raft_consensus.cc:729] T a2efd5e529db4df084288179f737a287 P c115200e338540ec9d442a2d072cf6cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c115200e338540ec9d442a2d072cf6cb, State: Initialized, Role: FOLLOWER
I20260812 05:24:13.421506 29507 consensus_queue.cc:260] T a2efd5e529db4df084288179f737a287 P 83e83f183de7453dafefae00cd3560fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e83f183de7453dafefae00cd3560fa" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 39183 } } peers { permanent_uuid: "c115200e338540ec9d442a2d072cf6cb" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 46537 } } peers { permanent_uuid: "c23f1ed3d1e747dbbcdcd9152af7718f" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 35489 } }
I20260812 05:24:13.421525 29508 consensus_queue.cc:260] T a2efd5e529db4df084288179f737a287 P c115200e338540ec9d442a2d072cf6cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e83f183de7453dafefae00cd3560fa" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 39183 } } peers { permanent_uuid: "c115200e338540ec9d442a2d072cf6cb" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 46537 } } peers { permanent_uuid: "c23f1ed3d1e747dbbcdcd9152af7718f" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 35489 } }
I20260812 05:24:13.421794 29507 ts_tablet_manager.cc:1428] T a2efd5e529db4df084288179f737a287 P 83e83f183de7453dafefae00cd3560fa: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260812 05:24:13.421790 29508 ts_tablet_manager.cc:1428] T a2efd5e529db4df084288179f737a287 P c115200e338540ec9d442a2d072cf6cb: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260812 05:24:13.421851 29271 heartbeater.cc:503] Master 127.31.100.62:41215 was elected leader, sending a full tablet report...
I20260812 05:24:13.421957 29032 heartbeater.cc:503] Master 127.31.100.62:41215 was elected leader, sending a full tablet report...
W20260812 05:24:13.457742 29274 tablet.cc:2369] T a2efd5e529db4df084288179f737a287 P c115200e338540ec9d442a2d072cf6cb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 05:24:13.518090 29516 raft_consensus.cc:482] T a2efd5e529db4df084288179f737a287 P c23f1ed3d1e747dbbcdcd9152af7718f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 05:24:13.518211 29516 raft_consensus.cc:504] T a2efd5e529db4df084288179f737a287 P c23f1ed3d1e747dbbcdcd9152af7718f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e83f183de7453dafefae00cd3560fa" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 39183 } } peers { permanent_uuid: "c115200e338540ec9d442a2d072cf6cb" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 46537 } } peers { permanent_uuid: "c23f1ed3d1e747dbbcdcd9152af7718f" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 35489 } }
I20260812 05:24:13.518589 29516 leader_election.cc:288] T a2efd5e529db4df084288179f737a287 P c23f1ed3d1e747dbbcdcd9152af7718f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 83e83f183de7453dafefae00cd3560fa (127.31.100.1:39183), c115200e338540ec9d442a2d072cf6cb (127.31.100.2:46537)
I20260812 05:24:13.521207 29499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:24:13.518801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.100.3:45543 (local address 127.31.100.1:39183)
0812 05:24:13.518852 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0812 05:24:13.518854 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 05:24:13.518866 (+ 12us) server_negotiation.cc:407] Connection header received
0812 05:24:13.518896 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:24:13.518896 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:24:13.518907 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:24:13.518964 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 05:24:13.519123 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:24:13.519624 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:24:13.520522 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:24:13.520895 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:24:13.520958 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:24:13.520968 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0812 05:24:13.521109 (+ 141us) server_negotiation.cc:299] Negotiation successful
0812 05:24:13.521149 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260812 05:24:13.521205 29481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:24:13.518738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.1:39183 (local address 127.31.100.3:45543)
0812 05:24:13.518790 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0812 05:24:13.518802 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0812 05:24:13.518842 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:24:13.518980 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:24:13.518981 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:24:13.518983 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 05:24:13.519060 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:24:13.519061 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:24:13.519699 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:24:13.519700 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:24:13.520435 (+ 735us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:24:13.520436 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:24:13.520968 (+ 532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:24:13.520969 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:24:13.521046 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:24:13.521050 (+ 4us) client_negotiation.cc:769] Sending connection context
0812 05:24:13.521093 (+ 43us) client_negotiation.cc:240] Negotiation successful
0812 05:24:13.521132 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260812 05:24:13.521468 29504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:24:13.518874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.100.3:55873 (local address 127.31.100.2:46537)
0812 05:24:13.518932 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0812 05:24:13.518934 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 05:24:13.519131 (+ 197us) server_negotiation.cc:407] Connection header received
0812 05:24:13.519165 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:24:13.519166 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:24:13.519176 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:24:13.519219 (+ 43us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 05:24:13.519371 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:24:13.519833 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:24:13.520730 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:24:13.521099 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:24:13.521168 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:24:13.521176 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0812 05:24:13.521336 (+ 160us) server_negotiation.cc:299] Negotiation successful
0812 05:24:13.521380 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260812 05:24:13.521567 28952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2efd5e529db4df084288179f737a287" candidate_uuid: "c23f1ed3d1e747dbbcdcd9152af7718f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83e83f183de7453dafefae00cd3560fa" is_pre_election: true
I20260812 05:24:13.521683 29523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:24:13.518744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.2:46537 (local address 127.31.100.3:55873)
0812 05:24:13.519059 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0812 05:24:13.519067 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0812 05:24:13.519117 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:24:13.519232 (+ 115us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:24:13.519233 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:24:13.519234 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 05:24:13.519306 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:24:13.519307 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:24:13.519907 (+ 600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:24:13.519908 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:24:13.520647 (+ 739us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:24:13.520648 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:24:13.521180 (+ 532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:24:13.521181 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:24:13.521247 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:24:13.521250 (+ 3us) client_negotiation.cc:769] Sending connection context
0812 05:24:13.521551 (+ 301us) client_negotiation.cc:240] Negotiation successful
0812 05:24:13.521594 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":298,"mutex_wait_us":139,"thread_start_us":116,"threads_started":1}
I20260812 05:24:13.521734 28952 raft_consensus.cc:2454] T a2efd5e529db4df084288179f737a287 P 83e83f183de7453dafefae00cd3560fa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c23f1ed3d1e747dbbcdcd9152af7718f in term 0.
I20260812 05:24:13.521939 29305 leader_election.cc:302] T a2efd5e529db4df084288179f737a287 P c23f1ed3d1e747dbbcdcd9152af7718f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83e83f183de7453dafefae00cd3560fa, c23f1ed3d1e747dbbcdcd9152af7718f; no voters:
I20260812 05:24:13.522014 29190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2efd5e529db4df084288179f737a287" candidate_uuid: "c23f1ed3d1e747dbbcdcd9152af7718f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c115200e338540ec9d442a2d072cf6cb" is_pre_election: true
I20260812 05:24:13.522064 29516 raft_consensus.cc:2781] T a2efd5e529db4df084288179f737a287 P c23f1ed3d1e747dbbcdcd9152af7718f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 05:24:13.522128 29516 raft_consensus.cc:482] T a2efd5e529db4df084288179f737a287 P c23f1ed3d1e747dbbcdcd9152af7718f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 05:24:13.522161 29516 raft_consensus.cc:3037] T a2efd5e529db4df084288179f737a287 P c23f1ed3d1e747dbbcdcd9152af7718f [term 0 FOLLOWER]: Advancing to term 1
I20260812 05:24:13.522185 29190 raft_consensus.cc:2454] T a2efd5e529db4df084288179f737a287 P c115200e338540ec9d442a2d072cf6cb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c23f1ed3d1e747dbbcdcd9152af7718f in term 0.
I20260812 05:24:13.522909 29516 raft_consensus.cc:504] T a2efd5e529db4df084288179f737a287 P c23f1ed3d1e747dbbcdcd9152af7718f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e83f183de7453dafefae00cd3560fa" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 39183 } } peers { permanent_uuid: "c115200e338540ec9d442a2d072cf6cb" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 46537 } } peers { permanent_uuid: "c23f1ed3d1e747dbbcdcd9152af7718f" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 35489 } }
I20260812 05:24:13.523018 29516 leader_election.cc:288] T a2efd5e529db4df084288179f737a287 P c23f1ed3d1e747dbbcdcd9152af7718f [CANDIDATE]: Term 1 election: Requested vote from peers 83e83f183de7453dafefae00cd3560fa (127.31.100.1:39183), c115200e338540ec9d442a2d072cf6cb (127.31.100.2:46537)
I20260812 05:24:13.523195 29190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2efd5e529db4df084288179f737a287" candidate_uuid: "c23f1ed3d1e747dbbcdcd9152af7718f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c115200e338540ec9d442a2d072cf6cb"
I20260812 05:24:13.523207 28952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2efd5e529db4df084288179f737a287" candidate_uuid: "c23f1ed3d1e747dbbcdcd9152af7718f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83e83f183de7453dafefae00cd3560fa"
I20260812 05:24:13.523265 29190 raft_consensus.cc:3037] T a2efd5e529db4df084288179f737a287 P c115200e338540ec9d442a2d072cf6cb [term 0 FOLLOWER]: Advancing to term 1
I20260812 05:24:13.523272 28952 raft_consensus.cc:3037] T a2efd5e529db4df084288179f737a287 P 83e83f183de7453dafefae00cd3560fa [term 0 FOLLOWER]: Advancing to term 1
I20260812 05:24:13.523979 28952 raft_consensus.cc:2454] T a2efd5e529db4df084288179f737a287 P 83e83f183de7453dafefae00cd3560fa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c23f1ed3d1e747dbbcdcd9152af7718f in term 1.
I20260812 05:24:13.524010 29190 raft_consensus.cc:2454] T a2efd5e529db4df084288179f737a287 P c115200e338540ec9d442a2d072cf6cb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c23f1ed3d1e747dbbcdcd9152af7718f in term 1.
I20260812 05:24:13.524143 29305 leader_election.cc:302] T a2efd5e529db4df084288179f737a287 P c23f1ed3d1e747dbbcdcd9152af7718f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83e83f183de7453dafefae00cd3560fa, c23f1ed3d1e747dbbcdcd9152af7718f; no voters:
I20260812 05:24:13.524243 29516 raft_consensus.cc:2781] T a2efd5e529db4df084288179f737a287 P c23f1ed3d1e747dbbcdcd9152af7718f [term 1 FOLLOWER]: Leader election won for term 1
I20260812 05:24:13.524374 29516 raft_consensus.cc:686] T a2efd5e529db4df084288179f737a287 P c23f1ed3d1e747dbbcdcd9152af7718f [term 1 LEADER]: Becoming Leader. State: Replica: c23f1ed3d1e747dbbcdcd9152af7718f, State: Running, Role: LEADER
I20260812 05:24:13.524448 29516 consensus_queue.cc:237] T a2efd5e529db4df084288179f737a287 P c23f1ed3d1e747dbbcdcd9152af7718f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e83f183de7453dafefae00cd3560fa" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 39183 } } peers { permanent_uuid: "c115200e338540ec9d442a2d072cf6cb" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 46537 } } peers { permanent_uuid: "c23f1ed3d1e747dbbcdcd9152af7718f" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 35489 } }
I20260812 05:24:13.525178 28690 catalog_manager.cc:5495] T a2efd5e529db4df084288179f737a287 P c23f1ed3d1e747dbbcdcd9152af7718f reported cstate change: term changed from 0 to 1, leader changed from <none> to c23f1ed3d1e747dbbcdcd9152af7718f (127.31.100.3). New cstate: current_term: 1 leader_uuid: "c23f1ed3d1e747dbbcdcd9152af7718f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e83f183de7453dafefae00cd3560fa" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 39183 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c115200e338540ec9d442a2d072cf6cb" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 46537 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c23f1ed3d1e747dbbcdcd9152af7718f" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 35489 } health_report { overall_health: HEALTHY } } }
W20260812 05:24:13.556886 29033 tablet.cc:2369] T a2efd5e529db4df084288179f737a287 P 83e83f183de7453dafefae00cd3560fa: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 05:24:13.557775 29503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:24:13.555448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40158 (local address 127.31.100.3:35489)
0812 05:24:13.555501 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0812 05:24:13.555503 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 05:24:13.555512 (+ 9us) server_negotiation.cc:407] Connection header received
0812 05:24:13.555592 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:24:13.555593 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:24:13.555610 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:24:13.555667 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 05:24:13.555777 (+ 110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:24:13.556234 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:24:13.556979 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:24:13.557193 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:24:13.557248 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:24:13.557406 (+ 158us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 05:24:13.557547 (+ 141us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 05:24:13.557621 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0812 05:24:13.557659 (+ 38us) server_negotiation.cc:299] Negotiation successful
0812 05:24:13.557679 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260812 05:24:13.559429 28952 raft_consensus.cc:1261] T a2efd5e529db4df084288179f737a287 P 83e83f183de7453dafefae00cd3560fa [term 1 FOLLOWER]: Refusing update from remote peer c23f1ed3d1e747dbbcdcd9152af7718f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 05:24:13.559429 29190 raft_consensus.cc:1261] T a2efd5e529db4df084288179f737a287 P c115200e338540ec9d442a2d072cf6cb [term 1 FOLLOWER]: Refusing update from remote peer c23f1ed3d1e747dbbcdcd9152af7718f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 05:24:13.559764 29524 consensus_queue.cc:1035] T a2efd5e529db4df084288179f737a287 P c23f1ed3d1e747dbbcdcd9152af7718f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c115200e338540ec9d442a2d072cf6cb" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 46537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 05:24:13.559885 29516 consensus_queue.cc:1035] T a2efd5e529db4df084288179f737a287 P c23f1ed3d1e747dbbcdcd9152af7718f [LEADER]: Connected to new peer: Peer: permanent_uuid: "83e83f183de7453dafefae00cd3560fa" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 39183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 05:24:13.564328 28689 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38572:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 05:24:13.564411 28689 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38572:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 05:24:13.565069 28689 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 38eaa403a01d42878a061129ceba1203: Sending DeleteTablet for 3 replicas of tablet a2efd5e529db4df084288179f737a287
I20260812 05:24:13.565376 29378 tablet_service.cc:1514] Processing DeleteTablet for tablet a2efd5e529db4df084288179f737a287 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 05:24:13 UTC) from {username='kudu', principal='kudu/127.31.100.62@KRBTEST.COM'} at 127.0.0.1:40154
I20260812 05:24:13.565366 29157 tablet_service.cc:1514] Processing DeleteTablet for tablet a2efd5e529db4df084288179f737a287 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 05:24:13 UTC) from {username='kudu', principal='kudu/127.31.100.62@KRBTEST.COM'} at 127.0.0.1:60140
I20260812 05:24:13.565663 32144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0waFvT/build/release/bin/kudu with pid 28809
I20260812 05:24:13.565717 28923 tablet_service.cc:1514] Processing DeleteTablet for tablet a2efd5e529db4df084288179f737a287 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 05:24:13 UTC) from {username='kudu', principal='kudu/127.31.100.62@KRBTEST.COM'} at 127.0.0.1:45658
I20260812 05:24:13.566171 29553 tablet_replica.cc:330] T a2efd5e529db4df084288179f737a287 P c115200e338540ec9d442a2d072cf6cb: stopping tablet replica
I20260812 05:24:13.566243 29553 raft_consensus.cc:2229] T a2efd5e529db4df084288179f737a287 P c115200e338540ec9d442a2d072cf6cb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 05:24:13.566310 29553 raft_consensus.cc:2258] T a2efd5e529db4df084288179f737a287 P c115200e338540ec9d442a2d072cf6cb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 05:24:13.566442 29555 tablet_replica.cc:330] T a2efd5e529db4df084288179f737a287 P c23f1ed3d1e747dbbcdcd9152af7718f: stopping tablet replica
I20260812 05:24:13.566516 29555 raft_consensus.cc:2229] T a2efd5e529db4df084288179f737a287 P c23f1ed3d1e747dbbcdcd9152af7718f [term 1 LEADER]: Raft consensus shutting down.
I20260812 05:24:13.566592 29555 raft_consensus.cc:2258] T a2efd5e529db4df084288179f737a287 P c23f1ed3d1e747dbbcdcd9152af7718f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 05:24:13.572649 29555 ts_tablet_manager.cc:1905] T a2efd5e529db4df084288179f737a287 P c23f1ed3d1e747dbbcdcd9152af7718f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 05:24:13.574049 29553 ts_tablet_manager.cc:1905] T a2efd5e529db4df084288179f737a287 P c115200e338540ec9d442a2d072cf6cb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 05:24:13.575613 29555 ts_tablet_manager.cc:1918] T a2efd5e529db4df084288179f737a287 P c23f1ed3d1e747dbbcdcd9152af7718f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 05:24:13.575700 29555 log.cc:1196] T a2efd5e529db4df084288179f737a287 P c23f1ed3d1e747dbbcdcd9152af7718f: Deleting WAL directory at /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786512017613366-32144-0/minicluster-data/ts-2/wal/wals/a2efd5e529db4df084288179f737a287
I20260812 05:24:13.575974 29555 ts_tablet_manager.cc:1939] T a2efd5e529db4df084288179f737a287 P c23f1ed3d1e747dbbcdcd9152af7718f: Deleting consensus metadata
I20260812 05:24:13.576450 29553 ts_tablet_manager.cc:1918] T a2efd5e529db4df084288179f737a287 P c115200e338540ec9d442a2d072cf6cb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 05:24:13.576529 29553 log.cc:1196] T a2efd5e529db4df084288179f737a287 P c115200e338540ec9d442a2d072cf6cb: Deleting WAL directory at /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786512017613366-32144-0/minicluster-data/ts-1/wal/wals/a2efd5e529db4df084288179f737a287
I20260812 05:24:13.576864 29553 ts_tablet_manager.cc:1939] T a2efd5e529db4df084288179f737a287 P c115200e338540ec9d442a2d072cf6cb: Deleting consensus metadata
I20260812 05:24:13.577288 28667 catalog_manager.cc:4841] TS c115200e338540ec9d442a2d072cf6cb (127.31.100.2:46537): tablet a2efd5e529db4df084288179f737a287 (table test-table [id=bfd02d1322a7417598cee7010251de76]) successfully deleted
I20260812 05:24:13.585356 28660 catalog_manager.cc:4841] TS c23f1ed3d1e747dbbcdcd9152af7718f (127.31.100.3:35489): tablet a2efd5e529db4df084288179f737a287 (table test-table [id=bfd02d1322a7417598cee7010251de76]) successfully deleted
W20260812 05:24:13.589483 28665 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.31.100.1:39183 (error 108)
I20260812 05:24:13.589972 29495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:24:13.589836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.1:39183 (local address 127.0.0.1:45664)
0812 05:24:13.589892 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0812 05:24:13.589938 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.100.1:39183: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260812 05:24:13.590058 32144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0waFvT/build/release/bin/kudu with pid 29038
W20260812 05:24:13.607240 28665 catalog_manager.cc:4584] TS 83e83f183de7453dafefae00cd3560fa (127.31.100.1:39183): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a2efd5e529db4df084288179f737a287: Network error: Client connection negotiation failed: client connection to 127.31.100.1:39183: connect: Connection refused (error 111)
I20260812 05:24:13.611399 32144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0waFvT/build/release/bin/kudu with pid 29279
I20260812 05:24:13.619899 32144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0waFvT/build/release/bin/kudu with pid 28625
2026-08-12T05:24:13Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2131 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-08-12T05:24:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T05:24:13Z Disabled control of system clock
I20260812 05:24:13.655381 32144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0waFvT/build/release/bin/kudu
/tmp/dist-test-task0waFvT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786512017613366-32144-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786512017613366-32144-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786512017613366-32144-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786512017613366-32144-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.100.62:40925
--webserver_interface=127.31.100.62
--webserver_port=0
--builtin_ntp_servers=127.31.100.20:43117
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.100.62:40925
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260812 05:24:13.733898 29567 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 05:24:13.734088 29567 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 05:24:13.734107 29567 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 05:24:13.735149 29567 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 05:24:13.735180 29567 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 05:24:13.735193 29567 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 05:24:13.735203 29567 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 05:24:13.735239 29567 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 05:24:13.736614 29567 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.100.20:43117
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786512017613366-32144-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786512017613366-32144-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.100.62:40925
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.100.62:40925
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786512017613366-32144-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.100.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29567
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786512017613366-32144-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 04:20:34 UTC on ccycloud.quasar-phqdqp.root.comops.site
build id 4581871
I20260812 05:24:13.736799 29567 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 05:24:13.737017 29567 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260812 05:24:13.741532 29567 server_base.cc:1034] running on GCE node
W20260812 05:24:13.741508 29585 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 05:24:13.741508 29582 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 05:24:13.741497 29581 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 05:24:13.741927 29567 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 05:24:13.742166 29567 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 05:24:13.743290 29567 hybrid_clock.cc:648] HybridClock initialized: now 1786512253743282 us; error 28 us; skew 500 ppm
I20260812 05:24:13.744390 29567 webserver.cc:460] Webserver started at http://127.31.100.62:33639/ using document root <none> and password file <none>
I20260812 05:24:13.744598 29567 fs_manager.cc:359] Metadata directory not provided
I20260812 05:24:13.744642 29567 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 05:24:13.744765 29567 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 05:24:13.745826 29567 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786512017613366-32144-0/minicluster-data/master-0/data/instance:
uuid: "052e677d293e4c7cb4487efcde0de98a"
format_stamp: "Formatted at 2026-08-12 05:24:13 on dist-test-slave-42z9"
server_key: "52d7358ea114a2a6a1b2131a0abd3cac"
server_key_iv: "1426959b7c6d4a9f422f1ae5d47a80e9"
server_key_version: "encryptionkey@0"
I20260812 05:24:13.746133 29567 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786512017613366-32144-0/minicluster-data/master-0/wal/instance:
uuid: "052e677d293e4c7cb4487efcde0de98a"
format_stamp: "Formatted at 2026-08-12 05:24:13 on dist-test-slave-42z9"
server_key: "52d7358ea114a2a6a1b2131a0abd3cac"
server_key_iv: "1426959b7c6d4a9f422f1ae5d47a80e9"
server_key_version: "encryptionkey@0"
I20260812 05:24:13.747262 29567 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260812 05:24:13.747900 29595 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 05:24:13.748044 29567 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260812 05:24:13.748126 29567 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786512017613366-32144-0/minicluster-data/master-0/data,/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786512017613366-32144-0/minicluster-data/master-0/wal
uuid: "052e677d293e4c7cb4487efcde0de98a"
format_stamp: "Formatted at 2026-08-12 05:24:13 on dist-test-slave-42z9"
server_key: "52d7358ea114a2a6a1b2131a0abd3cac"
server_key_iv: "1426959b7c6d4a9f422f1ae5d47a80e9"
server_key_version: "encryptionkey@0"
I20260812 05:24:13.748181 29567 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786512017613366-32144-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786512017613366-32144-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786512017613366-32144-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 05:24:13.767521 29567 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 05:24:13.767781 29567 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 05:24:13.767900 29567 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 05:24:13.771562 29567 rpc_server.cc:307] RPC server started. Bound to: 127.31.100.62:40925
I20260812 05:24:13.771597 29685 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.100.62:40925 every 8 connection(s)
I20260812 05:24:13.771880 29567 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786512017613366-32144-0/minicluster-data/master-0/data/info.pb
I20260812 05:24:13.772791 29686 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 05:24:13.775025 29686 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 052e677d293e4c7cb4487efcde0de98a: Bootstrap starting.
I20260812 05:24:13.775586 29686 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 052e677d293e4c7cb4487efcde0de98a: Neither blocks nor log segments found. Creating new log.
I20260812 05:24:13.775863 29686 log.cc:824] T 00000000000000000000000000000000 P 052e677d293e4c7cb4487efcde0de98a: Log is configured to *not* fsync() on all Append() calls
I20260812 05:24:13.776466 29686 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 052e677d293e4c7cb4487efcde0de98a: No bootstrap required, opened a new log
I20260812 05:24:13.778043 29686 raft_consensus.cc:348] T 00000000000000000000000000000000 P 052e677d293e4c7cb4487efcde0de98a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "052e677d293e4c7cb4487efcde0de98a" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 40925 } }
I20260812 05:24:13.778175 29686 raft_consensus.cc:374] T 00000000000000000000000000000000 P 052e677d293e4c7cb4487efcde0de98a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 05:24:13.778198 29686 raft_consensus.cc:729] T 00000000000000000000000000000000 P 052e677d293e4c7cb4487efcde0de98a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 052e677d293e4c7cb4487efcde0de98a, State: Initialized, Role: FOLLOWER
I20260812 05:24:13.778335 29686 consensus_queue.cc:260] T 00000000000000000000000000000000 P 052e677d293e4c7cb4487efcde0de98a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "052e677d293e4c7cb4487efcde0de98a" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 40925 } }
I20260812 05:24:13.778412 29686 raft_consensus.cc:388] T 00000000000000000000000000000000 P 052e677d293e4c7cb4487efcde0de98a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 05:24:13.778448 29686 raft_consensus.cc:482] T 00000000000000000000000000000000 P 052e677d293e4c7cb4487efcde0de98a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 05:24:13.778493 29686 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 052e677d293e4c7cb4487efcde0de98a [term 0 FOLLOWER]: Advancing to term 1
I20260812 05:24:13.778980 29686 raft_consensus.cc:504] T 00000000000000000000000000000000 P 052e677d293e4c7cb4487efcde0de98a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "052e677d293e4c7cb4487efcde0de98a" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 40925 } }
I20260812 05:24:13.779070 29686 leader_election.cc:302] T 00000000000000000000000000000000 P 052e677d293e4c7cb4487efcde0de98a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 052e677d293e4c7cb4487efcde0de98a; no voters:
I20260812 05:24:13.779219 29686 leader_election.cc:288] T 00000000000000000000000000000000 P 052e677d293e4c7cb4487efcde0de98a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 05:24:13.779268 29691 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 052e677d293e4c7cb4487efcde0de98a [term 1 FOLLOWER]: Leader election won for term 1
I20260812 05:24:13.779421 32144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0waFvT/build/release/bin/kudu as pid 29567
I20260812 05:24:13.779436 29691 raft_consensus.cc:686] T 00000000000000000000000000000000 P 052e677d293e4c7cb4487efcde0de98a [term 1 LEADER]: Becoming Leader. State: Replica: 052e677d293e4c7cb4487efcde0de98a, State: Running, Role: LEADER
I20260812 05:24:13.779488 29686 sys_catalog.cc:564] T 00000000000000000000000000000000 P 052e677d293e4c7cb4487efcde0de98a [sys.catalog]: configured and running, proceeding with master startup.
I20260812 05:24:13.779523 32144 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786512017613366-32144-0/minicluster-data/master-0/wal/instance
I20260812 05:24:13.779531 29691 consensus_queue.cc:237] T 00000000000000000000000000000000 P 052e677d293e4c7cb4487efcde0de98a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "052e677d293e4c7cb4487efcde0de98a" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 40925 } }
I20260812 05:24:13.779726 32144 external_mini_cluster.cc:1428] Setting key 78fd1fa48b3e888c8b98393020971686
I20260812 05:24:13.780114 29694 sys_catalog.cc:455] T 00000000000000000000000000000000 P 052e677d293e4c7cb4487efcde0de98a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 052e677d293e4c7cb4487efcde0de98a. Latest consensus state: current_term: 1 leader_uuid: "052e677d293e4c7cb4487efcde0de98a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "052e677d293e4c7cb4487efcde0de98a" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 40925 } } }
I20260812 05:24:13.780203 29694 sys_catalog.cc:458] T 00000000000000000000000000000000 P 052e677d293e4c7cb4487efcde0de98a [sys.catalog]: This master's current role is: LEADER
I20260812 05:24:13.780354 29692 sys_catalog.cc:455] T 00000000000000000000000000000000 P 052e677d293e4c7cb4487efcde0de98a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "052e677d293e4c7cb4487efcde0de98a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "052e677d293e4c7cb4487efcde0de98a" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 40925 } } }
I20260812 05:24:13.780438 29692 sys_catalog.cc:458] T 00000000000000000000000000000000 P 052e677d293e4c7cb4487efcde0de98a [sys.catalog]: This master's current role is: LEADER
I20260812 05:24:13.780819 29699 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 05:24:13.781342 29699 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 05:24:13.782862 29699 catalog_manager.cc:1347] Generated new cluster ID: c388f40bdc9a43adaf4dda3f3ff58e99
I20260812 05:24:13.782919 29699 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 05:24:13.786123 29700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:24:13.780593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41814 (local address 127.31.100.62:40925)
0812 05:24:13.780822 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0812 05:24:13.780830 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 05:24:13.780863 (+ 33us) server_negotiation.cc:407] Connection header received
0812 05:24:13.780973 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:24:13.780980 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:24:13.781056 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:24:13.781483 (+ 427us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:24:13.781572 (+ 89us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:24:13.782847 (+ 1275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:24:13.783942 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:24:13.784528 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:24:13.784918 (+ 390us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:24:13.785301 (+ 383us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 05:24:13.785304 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 05:24:13.785304 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 05:24:13.785520 (+ 216us) server_negotiation.cc:922] calling sasl_server_start()
0812 05:24:13.785558 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 05:24:13.785571 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 05:24:13.785619 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0812 05:24:13.785845 (+ 226us) server_negotiation.cc:299] Negotiation successful
0812 05:24:13.785916 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":59,"threads_started":1}
I20260812 05:24:13.793156 29699 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 05:24:13.793766 29699 catalog_manager.cc:1504] Loading token signing keys...
I20260812 05:24:13.801661 29699 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 052e677d293e4c7cb4487efcde0de98a: Generated new TSK 0
I20260812 05:24:13.801802 29699 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 05:24:13.807518 32144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0waFvT/build/release/bin/kudu
/tmp/dist-test-task0waFvT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786512017613366-32144-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786512017613366-32144-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786512017613366-32144-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786512017613366-32144-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.100.1:0
--local_ip_for_outbound_sockets=127.31.100.1
--webserver_interface=127.31.100.1
--webserver_port=0
--tserver_master_addrs=127.31.100.62:40925
--builtin_ntp_servers=127.31.100.20:43117
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260812 05:24:13.887902 29723 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 05:24:13.888090 29723 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 05:24:13.888119 29723 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 05:24:13.889223 29723 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 05:24:13.889261 29723 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 05:24:13.889293 29723 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.100.1
I20260812 05:24:13.890632 29723 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.100.20:43117
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786512017613366-32144-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786512017613366-32144-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.100.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786512017613366-32144-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.100.1
--webserver_port=0
--tserver_master_addrs=127.31.100.62:40925
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29723
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.100.1
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786512017613366-32144-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 04:20:34 UTC on ccycloud.quasar-phqdqp.root.comops.site
build id 4581871
I20260812 05:24:13.890859 29723 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 05:24:13.891120 29723 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 05:24:13.895869 29735 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 05:24:13.895911 29738 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 05:24:13.895887 29741 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 05:24:13.896128 29723 server_base.cc:1034] running on GCE node
I20260812 05:24:13.896317 29723 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 05:24:13.896531 29723 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 05:24:13.897680 29723 hybrid_clock.cc:648] HybridClock initialized: now 1786512253897669 us; error 31 us; skew 500 ppm
I20260812 05:24:13.898835 29723 webserver.cc:460] Webserver started at http://127.31.100.1:33307/ using document root <none> and password file <none>
I20260812 05:24:13.899113 29723 fs_manager.cc:359] Metadata directory not provided
I20260812 05:24:13.899166 29723 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 05:24:13.899296 29723 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 05:24:13.900302 29723 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786512017613366-32144-0/minicluster-data/ts-0/data/instance:
uuid: "80568930ffe64a73a23282e698c84509"
format_stamp: "Formatted at 2026-08-12 05:24:13 on dist-test-slave-42z9"
server_key: "7d23a8988ee635dbfb48b0455b2eb009"
server_key_iv: "f22862fa0fea307e085ece3e85f750ff"
server_key_version: "encryptionkey@0"
I20260812 05:24:13.900620 29723 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786512017613366-32144-0/minicluster-data/ts-0/wal/instance:
uuid: "80568930ffe64a73a23282e698c84509"
format_stamp: "Formatted at 2026-08-12 05:24:13 on dist-test-slave-42z9"
server_key: "7d23a8988ee635dbfb48b0455b2eb009"
server_key_iv: "f22862fa0fea307e085ece3e85f750ff"
server_key_version: "encryptionkey@0"
I20260812 05:24:13.901908 29723 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260812 05:24:13.902611 29749 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 05:24:13.902745 29723 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260812 05:24:13.902827 29723 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786512017613366-32144-0/minicluster-data/ts-0/data,/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786512017613366-32144-0/minicluster-data/ts-0/wal
uuid: "80568930ffe64a73a23282e698c84509"
format_stamp: "Formatted at 2026-08-12 05:24:13 on dist-test-slave-42z9"
server_key: "7d23a8988ee635dbfb48b0455b2eb009"
server_key_iv: "f22862fa0fea307e085ece3e85f750ff"
server_key_version: "encryptionkey@0"
I20260812 05:24:13.902886 29723 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786512017613366-32144-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786512017613366-32144-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786512017613366-32144-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 05:24:13.912868 29723 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 05:24:13.913107 29723 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 05:24:13.913220 29723 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 05:24:13.913411 29723 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 05:24:13.913720 29723 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 05:24:13.913753 29723 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 05:24:13.913779 29723 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 05:24:13.913805 29723 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 05:24:13.919075 29723 rpc_server.cc:307] RPC server started. Bound to: 127.31.100.1:41313
I20260812 05:24:13.919112 29933 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.100.1:41313 every 8 connection(s)
I20260812 05:24:13.919394 29723 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786512017613366-32144-0/minicluster-data/ts-0/data/info.pb
I20260812 05:24:13.922847 32144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0waFvT/build/release/bin/kudu as pid 29723
I20260812 05:24:13.922926 32144 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786512017613366-32144-0/minicluster-data/ts-0/wal/instance
I20260812 05:24:13.923090 32144 external_mini_cluster.cc:1428] Setting key 570982b2a4cc1ff1d1629a6f71049a23
I20260812 05:24:13.923921 32144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0waFvT/build/release/bin/kudu
/tmp/dist-test-task0waFvT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786512017613366-32144-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786512017613366-32144-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786512017613366-32144-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786512017613366-32144-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.100.2:0
--local_ip_for_outbound_sockets=127.31.100.2
--webserver_interface=127.31.100.2
--webserver_port=0
--tserver_master_addrs=127.31.100.62:40925
--builtin_ntp_servers=127.31.100.20:43117
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260812 05:24:13.924245 29700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:24:13.919995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.100.1:40193 (local address 127.31.100.62:40925)
0812 05:24:13.920035 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0812 05:24:13.920037 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 05:24:13.920240 (+ 203us) server_negotiation.cc:407] Connection header received
0812 05:24:13.920698 (+ 458us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:24:13.920699 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:24:13.920714 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:24:13.920764 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:24:13.921372 (+ 608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:24:13.921921 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:24:13.922838 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:24:13.923103 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:24:13.923167 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:24:13.923964 (+ 797us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 05:24:13.923966 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 05:24:13.923966 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 05:24:13.923971 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0812 05:24:13.923988 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 05:24:13.923993 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 05:24:13.924049 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0812 05:24:13.924164 (+ 115us) server_negotiation.cc:299] Negotiation successful
0812 05:24:13.924190 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260812 05:24:13.924433 29937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:24:13.919974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.62:40925 (local address 127.31.100.1:40193)
0812 05:24:13.920135 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0812 05:24:13.920165 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0812 05:24:13.920607 (+ 442us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:24:13.920826 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:24:13.920830 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:24:13.920840 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0812 05:24:13.921288 (+ 448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:24:13.921292 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:24:13.922008 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:24:13.922009 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:24:13.922755 (+ 746us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:24:13.922757 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:24:13.923197 (+ 440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:24:13.923198 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:24:13.923352 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:24:13.923833 (+ 481us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 05:24:13.923834 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0812 05:24:13.923860 (+ 26us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 05:24:13.923880 (+ 20us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 05:24:13.924079 (+ 199us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 05:24:13.924086 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 05:24:13.924089 (+ 3us) client_negotiation.cc:769] Sending connection context
0812 05:24:13.924152 (+ 63us) client_negotiation.cc:240] Negotiation successful
0812 05:24:13.924249 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":69,"threads_started":1}
I20260812 05:24:13.924913 29934 heartbeater.cc:347] Connected to a master server at 127.31.100.62:40925
I20260812 05:24:13.924998 29934 heartbeater.cc:464] Registering TS with master...
I20260812 05:24:13.925173 29934 heartbeater.cc:511] Master 127.31.100.62:40925 requested a full tablet report, sending...
I20260812 05:24:13.925694 29623 ts_manager.cc:194] Registered new tserver with Master: 80568930ffe64a73a23282e698c84509 (127.31.100.1:41313)
I20260812 05:24:13.926494 29623 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.31.100.1:40193
W20260812 05:24:14.004279 29938 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 05:24:14.004467 29938 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 05:24:14.004487 29938 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 05:24:14.005649 29938 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 05:24:14.005688 29938 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 05:24:14.005712 29938 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.100.2
I20260812 05:24:14.007030 29938 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.100.20:43117
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786512017613366-32144-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786512017613366-32144-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.100.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786512017613366-32144-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.100.2
--webserver_port=0
--tserver_master_addrs=127.31.100.62:40925
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29938
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.100.2
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786512017613366-32144-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 04:20:34 UTC on ccycloud.quasar-phqdqp.root.comops.site
build id 4581871
I20260812 05:24:14.007207 29938 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 05:24:14.007433 29938 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 05:24:14.012097 29955 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 05:24:14.012118 29949 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 05:24:14.012126 29938 server_base.cc:1034] running on GCE node
W20260812 05:24:14.012109 29946 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 05:24:14.012473 29938 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 05:24:14.012678 29938 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 05:24:14.013803 29938 hybrid_clock.cc:648] HybridClock initialized: now 1786512254013793 us; error 28 us; skew 500 ppm
I20260812 05:24:14.014894 29938 webserver.cc:460] Webserver started at http://127.31.100.2:44365/ using document root <none> and password file <none>
I20260812 05:24:14.015102 29938 fs_manager.cc:359] Metadata directory not provided
I20260812 05:24:14.015144 29938 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 05:24:14.015259 29938 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 05:24:14.016223 29938 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786512017613366-32144-0/minicluster-data/ts-1/data/instance:
uuid: "69da380f117345049a555d67d92d9238"
format_stamp: "Formatted at 2026-08-12 05:24:14 on dist-test-slave-42z9"
server_key: "97b72c83263d03156c9317cd81c9fe19"
server_key_iv: "d209d50088c726bb1e9482582023ec4a"
server_key_version: "encryptionkey@0"
I20260812 05:24:14.016530 29938 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786512017613366-32144-0/minicluster-data/ts-1/wal/instance:
uuid: "69da380f117345049a555d67d92d9238"
format_stamp: "Formatted at 2026-08-12 05:24:14 on dist-test-slave-42z9"
server_key: "97b72c83263d03156c9317cd81c9fe19"
server_key_iv: "d209d50088c726bb1e9482582023ec4a"
server_key_version: "encryptionkey@0"
I20260812 05:24:14.017791 29938 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260812 05:24:14.018435 29964 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 05:24:14.018595 29938 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260812 05:24:14.018677 29938 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786512017613366-32144-0/minicluster-data/ts-1/data,/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786512017613366-32144-0/minicluster-data/ts-1/wal
uuid: "69da380f117345049a555d67d92d9238"
format_stamp: "Formatted at 2026-08-12 05:24:14 on dist-test-slave-42z9"
server_key: "97b72c83263d03156c9317cd81c9fe19"
server_key_iv: "d209d50088c726bb1e9482582023ec4a"
server_key_version: "encryptionkey@0"
I20260812 05:24:14.018736 29938 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786512017613366-32144-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786512017613366-32144-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786512017613366-32144-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 05:24:14.047524 29938 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 05:24:14.047813 29938 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 05:24:14.047948 29938 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 05:24:14.048193 29938 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 05:24:14.048511 29938 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 05:24:14.048547 29938 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 05:24:14.048576 29938 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 05:24:14.048594 29938 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 05:24:14.054008 29938 rpc_server.cc:307] RPC server started. Bound to: 127.31.100.2:44947
I20260812 05:24:14.054078 30164 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.100.2:44947 every 8 connection(s)
I20260812 05:24:14.054363 29938 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786512017613366-32144-0/minicluster-data/ts-1/data/info.pb
I20260812 05:24:14.059643 32144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0waFvT/build/release/bin/kudu as pid 29938
I20260812 05:24:14.059721 32144 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786512017613366-32144-0/minicluster-data/ts-1/wal/instance
I20260812 05:24:14.059887 32144 external_mini_cluster.cc:1428] Setting key bd9d06a90c17293f46b93de7abe3d433
I20260812 05:24:14.060832 32144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0waFvT/build/release/bin/kudu
/tmp/dist-test-task0waFvT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786512017613366-32144-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786512017613366-32144-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786512017613366-32144-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786512017613366-32144-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.100.3:0
--local_ip_for_outbound_sockets=127.31.100.3
--webserver_interface=127.31.100.3
--webserver_port=0
--tserver_master_addrs=127.31.100.62:40925
--builtin_ntp_servers=127.31.100.20:43117
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260812 05:24:14.061123 29700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:24:14.057057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.100.2:51611 (local address 127.31.100.62:40925)
0812 05:24:14.057143 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0812 05:24:14.057145 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 05:24:14.057214 (+ 69us) server_negotiation.cc:407] Connection header received
0812 05:24:14.057710 (+ 496us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:24:14.057711 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:24:14.057721 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:24:14.057771 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:24:14.058373 (+ 602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:24:14.058859 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:24:14.059802 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:24:14.060024 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:24:14.060086 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:24:14.060854 (+ 768us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 05:24:14.060856 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 05:24:14.060857 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 05:24:14.060861 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0812 05:24:14.060876 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 05:24:14.060881 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 05:24:14.060927 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0812 05:24:14.061008 (+ 81us) server_negotiation.cc:299] Negotiation successful
0812 05:24:14.061042 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260812 05:24:14.061334 30169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:24:14.054989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.62:40925 (local address 127.31.100.2:51611)
0812 05:24:14.057103 (+ 2114us) negotiation.cc:107] Waiting for socket to connect
0812 05:24:14.057148 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0812 05:24:14.057629 (+ 481us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:24:14.057823 (+ 194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:24:14.057826 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:24:14.057834 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0812 05:24:14.058315 (+ 481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:24:14.058319 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:24:14.058918 (+ 599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:24:14.058918 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:24:14.059721 (+ 803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:24:14.059724 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:24:14.060095 (+ 371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:24:14.060097 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:24:14.060231 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:24:14.060721 (+ 490us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 05:24:14.060721 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0812 05:24:14.060749 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 05:24:14.060771 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 05:24:14.060932 (+ 161us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 05:24:14.060937 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 05:24:14.060940 (+ 3us) client_negotiation.cc:769] Sending connection context
0812 05:24:14.061012 (+ 72us) client_negotiation.cc:240] Negotiation successful
0812 05:24:14.061120 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2038,"thread_start_us":1978,"threads_started":1}
I20260812 05:24:14.061818 30165 heartbeater.cc:347] Connected to a master server at 127.31.100.62:40925
I20260812 05:24:14.061921 30165 heartbeater.cc:464] Registering TS with master...
I20260812 05:24:14.062101 30165 heartbeater.cc:511] Master 127.31.100.62:40925 requested a full tablet report, sending...
I20260812 05:24:14.062479 29623 ts_manager.cc:194] Registered new tserver with Master: 69da380f117345049a555d67d92d9238 (127.31.100.2:44947)
I20260812 05:24:14.062911 29623 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.31.100.2:51611
W20260812 05:24:14.142067 30171 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 05:24:14.142261 30171 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 05:24:14.142288 30171 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 05:24:14.143376 30171 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 05:24:14.143421 30171 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 05:24:14.143455 30171 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.100.3
I20260812 05:24:14.144754 30171 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.100.20:43117
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786512017613366-32144-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786512017613366-32144-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.100.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786512017613366-32144-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.100.3
--webserver_port=0
--tserver_master_addrs=127.31.100.62:40925
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30171
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.100.3
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786512017613366-32144-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 04:20:34 UTC on ccycloud.quasar-phqdqp.root.comops.site
build id 4581871
I20260812 05:24:14.144996 30171 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 05:24:14.145267 30171 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 05:24:14.150069 30180 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 05:24:14.150218 30187 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 05:24:14.150151 30183 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 05:24:14.150074 30171 server_base.cc:1034] running on GCE node
I20260812 05:24:14.150480 30171 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 05:24:14.150717 30171 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 05:24:14.151863 30171 hybrid_clock.cc:648] HybridClock initialized: now 1786512254151848 us; error 29 us; skew 500 ppm
I20260812 05:24:14.153087 30171 webserver.cc:460] Webserver started at http://127.31.100.3:33063/ using document root <none> and password file <none>
I20260812 05:24:14.153319 30171 fs_manager.cc:359] Metadata directory not provided
I20260812 05:24:14.153368 30171 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 05:24:14.153543 30171 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 05:24:14.154584 30171 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786512017613366-32144-0/minicluster-data/ts-2/data/instance:
uuid: "9271647c0cc4457db2e58ea0cc2efdb9"
format_stamp: "Formatted at 2026-08-12 05:24:14 on dist-test-slave-42z9"
server_key: "9da77c093f4371fe7e31b1f67d728b3c"
server_key_iv: "38f138c3c397987c1bee5458e038b157"
server_key_version: "encryptionkey@0"
I20260812 05:24:14.154903 30171 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786512017613366-32144-0/minicluster-data/ts-2/wal/instance:
uuid: "9271647c0cc4457db2e58ea0cc2efdb9"
format_stamp: "Formatted at 2026-08-12 05:24:14 on dist-test-slave-42z9"
server_key: "9da77c093f4371fe7e31b1f67d728b3c"
server_key_iv: "38f138c3c397987c1bee5458e038b157"
server_key_version: "encryptionkey@0"
I20260812 05:24:14.156122 30171 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260812 05:24:14.156761 30194 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 05:24:14.156906 30171 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260812 05:24:14.156996 30171 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786512017613366-32144-0/minicluster-data/ts-2/data,/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786512017613366-32144-0/minicluster-data/ts-2/wal
uuid: "9271647c0cc4457db2e58ea0cc2efdb9"
format_stamp: "Formatted at 2026-08-12 05:24:14 on dist-test-slave-42z9"
server_key: "9da77c093f4371fe7e31b1f67d728b3c"
server_key_iv: "38f138c3c397987c1bee5458e038b157"
server_key_version: "encryptionkey@0"
I20260812 05:24:14.157060 30171 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786512017613366-32144-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786512017613366-32144-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786512017613366-32144-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 05:24:14.175499 30171 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 05:24:14.175845 30171 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 05:24:14.175981 30171 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 05:24:14.176178 30171 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 05:24:14.176497 30171 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 05:24:14.176532 30171 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 05:24:14.176553 30171 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 05:24:14.176571 30171 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 05:24:14.182327 30171 rpc_server.cc:307] RPC server started. Bound to: 127.31.100.3:32799
I20260812 05:24:14.182379 30391 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.100.3:32799 every 8 connection(s)
I20260812 05:24:14.182682 30171 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786512017613366-32144-0/minicluster-data/ts-2/data/info.pb
I20260812 05:24:14.186506 32144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0waFvT/build/release/bin/kudu as pid 30171
I20260812 05:24:14.186595 32144 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786512017613366-32144-0/minicluster-data/ts-2/wal/instance
I20260812 05:24:14.186808 32144 external_mini_cluster.cc:1428] Setting key b78d562315695bd4541b9bdc5758a116
I20260812 05:24:14.187495 29700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:24:14.183323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.100.3:43299 (local address 127.31.100.62:40925)
0812 05:24:14.183371 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0812 05:24:14.183373 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 05:24:14.183613 (+ 240us) server_negotiation.cc:407] Connection header received
0812 05:24:14.184089 (+ 476us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:24:14.184091 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:24:14.184101 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:24:14.184139 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:24:14.184733 (+ 594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:24:14.185214 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:24:14.186128 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:24:14.186310 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:24:14.186355 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:24:14.187233 (+ 878us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 05:24:14.187234 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 05:24:14.187234 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 05:24:14.187238 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0812 05:24:14.187253 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 05:24:14.187256 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 05:24:14.187295 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0812 05:24:14.187384 (+ 89us) server_negotiation.cc:299] Negotiation successful
0812 05:24:14.187415 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260812 05:24:14.187685 30394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:24:14.183316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.62:40925 (local address 127.31.100.3:43299)
0812 05:24:14.183502 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0812 05:24:14.183534 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0812 05:24:14.184002 (+ 468us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:24:14.184191 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:24:14.184194 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:24:14.184201 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0812 05:24:14.184675 (+ 474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:24:14.184680 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:24:14.185269 (+ 589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:24:14.185270 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:24:14.186068 (+ 798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:24:14.186071 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:24:14.186375 (+ 304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:24:14.186376 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:24:14.186529 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:24:14.187063 (+ 534us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 05:24:14.187064 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0812 05:24:14.187092 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 05:24:14.187122 (+ 30us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 05:24:14.187306 (+ 184us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 05:24:14.187313 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 05:24:14.187316 (+ 3us) client_negotiation.cc:769] Sending connection context
0812 05:24:14.187380 (+ 64us) client_negotiation.cc:240] Negotiation successful
0812 05:24:14.187480 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":86,"threads_started":1}
I20260812 05:24:14.188068 30392 heartbeater.cc:347] Connected to a master server at 127.31.100.62:40925
I20260812 05:24:14.188150 30392 heartbeater.cc:464] Registering TS with master...
I20260812 05:24:14.188314 30392 heartbeater.cc:511] Master 127.31.100.62:40925 requested a full tablet report, sending...
I20260812 05:24:14.188643 29623 ts_manager.cc:194] Registered new tserver with Master: 9271647c0cc4457db2e58ea0cc2efdb9 (127.31.100.3:32799)
I20260812 05:24:14.189002 29623 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.31.100.3:43299
I20260812 05:24:14.198442 32144 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 05:24:14.202185 29700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:24:14.199756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41816 (local address 127.31.100.62:40925)
0812 05:24:14.199821 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0812 05:24:14.199823 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 05:24:14.199885 (+ 62us) server_negotiation.cc:407] Connection header received
0812 05:24:14.199925 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:24:14.199926 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:24:14.199936 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:24:14.199980 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:24:14.200109 (+ 129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:24:14.200581 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:24:14.201240 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:24:14.201406 (+ 166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:24:14.201486 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:24:14.201992 (+ 506us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 05:24:14.201993 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 05:24:14.201993 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 05:24:14.201997 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0812 05:24:14.202011 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 05:24:14.202015 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 05:24:14.202053 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0812 05:24:14.202084 (+ 31us) server_negotiation.cc:299] Negotiation successful
0812 05:24:14.202111 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260812 05:24:14.203472 29623 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41816:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 05:24:14.204967 29623 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 05:24:14.210839 30416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:24:14.207174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.2:44947 (local address 127.0.0.1:55300)
0812 05:24:14.207289 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0812 05:24:14.207299 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0812 05:24:14.207348 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:24:14.207730 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:24:14.207731 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:24:14.207733 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 05:24:14.207760 (+ 27us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:24:14.207761 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:24:14.209001 (+ 1240us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:24:14.209002 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:24:14.209899 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:24:14.209901 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:24:14.210564 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:24:14.210566 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:24:14.210645 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:24:14.210658 (+ 13us) client_negotiation.cc:769] Sending connection context
0812 05:24:14.210720 (+ 62us) client_negotiation.cc:240] Negotiation successful
0812 05:24:14.210768 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":97,"mutex_wait_us":9,"thread_start_us":70,"threads_started":1}
I20260812 05:24:14.210848 30417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:24:14.207385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55300 (local address 127.31.100.2:44947)
0812 05:24:14.207575 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0812 05:24:14.207579 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 05:24:14.207587 (+ 8us) server_negotiation.cc:407] Connection header received
0812 05:24:14.207622 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:24:14.207624 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:24:14.207637 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:24:14.207827 (+ 190us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 05:24:14.207883 (+ 56us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:24:14.208945 (+ 1062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:24:14.209985 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:24:14.210488 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:24:14.210551 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:24:14.210641 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0812 05:24:14.210733 (+ 92us) server_negotiation.cc:299] Negotiation successful
0812 05:24:14.210771 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":157,"thread_start_us":100,"threads_started":1}
I20260812 05:24:14.211027 30409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:24:14.206966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.1:41313 (local address 127.0.0.1:59252)
0812 05:24:14.207224 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0812 05:24:14.207234 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0812 05:24:14.207273 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:24:14.207642 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:24:14.207643 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:24:14.207645 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 05:24:14.207690 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:24:14.207692 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:24:14.208966 (+ 1274us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:24:14.208967 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:24:14.209983 (+ 1016us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:24:14.209985 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:24:14.210797 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:24:14.210798 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:24:14.210864 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:24:14.210868 (+ 4us) client_negotiation.cc:769] Sending connection context
0812 05:24:14.210919 (+ 51us) client_negotiation.cc:240] Negotiation successful
0812 05:24:14.210959 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":95,"threads_started":1}
I20260812 05:24:14.211344 30414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:24:14.207058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59252 (local address 127.31.100.1:41313)
0812 05:24:14.207417 (+ 359us) server_negotiation.cc:206] Beginning negotiation
0812 05:24:14.207421 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 05:24:14.207430 (+ 9us) server_negotiation.cc:407] Connection header received
0812 05:24:14.207466 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:24:14.207467 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:24:14.207479 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:24:14.207548 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 05:24:14.207765 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:24:14.208882 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:24:14.210050 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:24:14.210723 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:24:14.210784 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:24:14.210862 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0812 05:24:14.211232 (+ 370us) server_negotiation.cc:299] Negotiation successful
0812 05:24:14.211270 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":323,"thread_start_us":81,"threads_started":1}
I20260812 05:24:14.211398 30413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:24:14.206962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.3:32799 (local address 127.0.0.1:39020)
0812 05:24:14.207141 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0812 05:24:14.207181 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0812 05:24:14.207246 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:24:14.207553 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:24:14.207556 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:24:14.207559 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 05:24:14.207618 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:24:14.207620 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:24:14.208528 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:24:14.208529 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:24:14.209723 (+ 1194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:24:14.209725 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:24:14.211172 (+ 1447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:24:14.211173 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:24:14.211237 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:24:14.211240 (+ 3us) client_negotiation.cc:769] Sending connection context
0812 05:24:14.211291 (+ 51us) client_negotiation.cc:240] Negotiation successful
0812 05:24:14.211327 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":73,"threads_started":1}
I20260812 05:24:14.211421 30415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:24:14.207077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39020 (local address 127.31.100.3:32799)
0812 05:24:14.207373 (+ 296us) server_negotiation.cc:206] Beginning negotiation
0812 05:24:14.207378 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 05:24:14.207386 (+ 8us) server_negotiation.cc:407] Connection header received
0812 05:24:14.207420 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:24:14.207422 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:24:14.207437 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:24:14.207690 (+ 253us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 05:24:14.207740 (+ 50us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:24:14.208444 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:24:14.209796 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:24:14.210484 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:24:14.210570 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:24:14.210649 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0812 05:24:14.211309 (+ 660us) server_negotiation.cc:299] Negotiation successful
0812 05:24:14.211348 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":261,"thread_start_us":76,"threads_started":1}
I20260812 05:24:14.211633 30053 tablet_service.cc:1467] Processing CreateTablet for tablet bb7ac396618d42f4a0fa1d9ba52e7b6d (DEFAULT_TABLE table=test-table [id=a9f4e6fd99cc471397f7751af8c483f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 05:24:14.212008 30053 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb7ac396618d42f4a0fa1d9ba52e7b6d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 05:24:14.212101 30277 tablet_service.cc:1467] Processing CreateTablet for tablet bb7ac396618d42f4a0fa1d9ba52e7b6d (DEFAULT_TABLE table=test-table [id=a9f4e6fd99cc471397f7751af8c483f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 05:24:14.212284 29824 tablet_service.cc:1467] Processing CreateTablet for tablet bb7ac396618d42f4a0fa1d9ba52e7b6d (DEFAULT_TABLE table=test-table [id=a9f4e6fd99cc471397f7751af8c483f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 05:24:14.212436 30277 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb7ac396618d42f4a0fa1d9ba52e7b6d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 05:24:14.212594 29824 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb7ac396618d42f4a0fa1d9ba52e7b6d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 05:24:14.214810 30418 tablet_bootstrap.cc:492] T bb7ac396618d42f4a0fa1d9ba52e7b6d P 69da380f117345049a555d67d92d9238: Bootstrap starting.
I20260812 05:24:14.215397 30418 tablet_bootstrap.cc:654] T bb7ac396618d42f4a0fa1d9ba52e7b6d P 69da380f117345049a555d67d92d9238: Neither blocks nor log segments found. Creating new log.
I20260812 05:24:14.215667 30419 tablet_bootstrap.cc:492] T bb7ac396618d42f4a0fa1d9ba52e7b6d P 9271647c0cc4457db2e58ea0cc2efdb9: Bootstrap starting.
I20260812 05:24:14.215683 30420 tablet_bootstrap.cc:492] T bb7ac396618d42f4a0fa1d9ba52e7b6d P 80568930ffe64a73a23282e698c84509: Bootstrap starting.
I20260812 05:24:14.215752 30418 log.cc:824] T bb7ac396618d42f4a0fa1d9ba52e7b6d P 69da380f117345049a555d67d92d9238: Log is configured to *not* fsync() on all Append() calls
I20260812 05:24:14.216347 30420 tablet_bootstrap.cc:654] T bb7ac396618d42f4a0fa1d9ba52e7b6d P 80568930ffe64a73a23282e698c84509: Neither blocks nor log segments found. Creating new log.
I20260812 05:24:14.216347 30419 tablet_bootstrap.cc:654] T bb7ac396618d42f4a0fa1d9ba52e7b6d P 9271647c0cc4457db2e58ea0cc2efdb9: Neither blocks nor log segments found. Creating new log.
I20260812 05:24:14.216661 30418 tablet_bootstrap.cc:492] T bb7ac396618d42f4a0fa1d9ba52e7b6d P 69da380f117345049a555d67d92d9238: No bootstrap required, opened a new log
I20260812 05:24:14.216683 30419 log.cc:824] T bb7ac396618d42f4a0fa1d9ba52e7b6d P 9271647c0cc4457db2e58ea0cc2efdb9: Log is configured to *not* fsync() on all Append() calls
I20260812 05:24:14.216724 30418 ts_tablet_manager.cc:1397] T bb7ac396618d42f4a0fa1d9ba52e7b6d P 69da380f117345049a555d67d92d9238: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260812 05:24:14.216722 30420 log.cc:824] T bb7ac396618d42f4a0fa1d9ba52e7b6d P 80568930ffe64a73a23282e698c84509: Log is configured to *not* fsync() on all Append() calls
I20260812 05:24:14.217356 30419 tablet_bootstrap.cc:492] T bb7ac396618d42f4a0fa1d9ba52e7b6d P 9271647c0cc4457db2e58ea0cc2efdb9: No bootstrap required, opened a new log
I20260812 05:24:14.217365 30420 tablet_bootstrap.cc:492] T bb7ac396618d42f4a0fa1d9ba52e7b6d P 80568930ffe64a73a23282e698c84509: No bootstrap required, opened a new log
I20260812 05:24:14.217428 30420 ts_tablet_manager.cc:1397] T bb7ac396618d42f4a0fa1d9ba52e7b6d P 80568930ffe64a73a23282e698c84509: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260812 05:24:14.217428 30419 ts_tablet_manager.cc:1397] T bb7ac396618d42f4a0fa1d9ba52e7b6d P 9271647c0cc4457db2e58ea0cc2efdb9: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260812 05:24:14.218593 30418 raft_consensus.cc:348] T bb7ac396618d42f4a0fa1d9ba52e7b6d P 69da380f117345049a555d67d92d9238 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9271647c0cc4457db2e58ea0cc2efdb9" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 32799 } } peers { permanent_uuid: "80568930ffe64a73a23282e698c84509" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 41313 } } peers { permanent_uuid: "69da380f117345049a555d67d92d9238" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 44947 } }
I20260812 05:24:14.218722 30418 raft_consensus.cc:374] T bb7ac396618d42f4a0fa1d9ba52e7b6d P 69da380f117345049a555d67d92d9238 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 05:24:14.218753 30418 raft_consensus.cc:729] T bb7ac396618d42f4a0fa1d9ba52e7b6d P 69da380f117345049a555d67d92d9238 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69da380f117345049a555d67d92d9238, State: Initialized, Role: FOLLOWER
I20260812 05:24:14.218881 30418 consensus_queue.cc:260] T bb7ac396618d42f4a0fa1d9ba52e7b6d P 69da380f117345049a555d67d92d9238 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9271647c0cc4457db2e58ea0cc2efdb9" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 32799 } } peers { permanent_uuid: "80568930ffe64a73a23282e698c84509" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 41313 } } peers { permanent_uuid: "69da380f117345049a555d67d92d9238" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 44947 } }
I20260812 05:24:14.219125 30418 ts_tablet_manager.cc:1428] T bb7ac396618d42f4a0fa1d9ba52e7b6d P 69da380f117345049a555d67d92d9238: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260812 05:24:14.219245 30165 heartbeater.cc:503] Master 127.31.100.62:40925 was elected leader, sending a full tablet report...
I20260812 05:24:14.219779 30419 raft_consensus.cc:348] T bb7ac396618d42f4a0fa1d9ba52e7b6d P 9271647c0cc4457db2e58ea0cc2efdb9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9271647c0cc4457db2e58ea0cc2efdb9" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 32799 } } peers { permanent_uuid: "80568930ffe64a73a23282e698c84509" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 41313 } } peers { permanent_uuid: "69da380f117345049a555d67d92d9238" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 44947 } }
I20260812 05:24:14.219929 30419 raft_consensus.cc:374] T bb7ac396618d42f4a0fa1d9ba52e7b6d P 9271647c0cc4457db2e58ea0cc2efdb9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 05:24:14.219967 30419 raft_consensus.cc:729] T bb7ac396618d42f4a0fa1d9ba52e7b6d P 9271647c0cc4457db2e58ea0cc2efdb9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9271647c0cc4457db2e58ea0cc2efdb9, State: Initialized, Role: FOLLOWER
I20260812 05:24:14.220079 30419 consensus_queue.cc:260] T bb7ac396618d42f4a0fa1d9ba52e7b6d P 9271647c0cc4457db2e58ea0cc2efdb9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9271647c0cc4457db2e58ea0cc2efdb9" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 32799 } } peers { permanent_uuid: "80568930ffe64a73a23282e698c84509" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 41313 } } peers { permanent_uuid: "69da380f117345049a555d67d92d9238" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 44947 } }
I20260812 05:24:14.220135 30420 raft_consensus.cc:348] T bb7ac396618d42f4a0fa1d9ba52e7b6d P 80568930ffe64a73a23282e698c84509 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9271647c0cc4457db2e58ea0cc2efdb9" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 32799 } } peers { permanent_uuid: "80568930ffe64a73a23282e698c84509" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 41313 } } peers { permanent_uuid: "69da380f117345049a555d67d92d9238" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 44947 } }
I20260812 05:24:14.220266 30420 raft_consensus.cc:374] T bb7ac396618d42f4a0fa1d9ba52e7b6d P 80568930ffe64a73a23282e698c84509 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 05:24:14.220297 30420 raft_consensus.cc:729] T bb7ac396618d42f4a0fa1d9ba52e7b6d P 80568930ffe64a73a23282e698c84509 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80568930ffe64a73a23282e698c84509, State: Initialized, Role: FOLLOWER
I20260812 05:24:14.220320 30419 ts_tablet_manager.cc:1428] T bb7ac396618d42f4a0fa1d9ba52e7b6d P 9271647c0cc4457db2e58ea0cc2efdb9: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260812 05:24:14.220355 30392 heartbeater.cc:503] Master 127.31.100.62:40925 was elected leader, sending a full tablet report...
I20260812 05:24:14.220415 30420 consensus_queue.cc:260] T bb7ac396618d42f4a0fa1d9ba52e7b6d P 80568930ffe64a73a23282e698c84509 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9271647c0cc4457db2e58ea0cc2efdb9" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 32799 } } peers { permanent_uuid: "80568930ffe64a73a23282e698c84509" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 41313 } } peers { permanent_uuid: "69da380f117345049a555d67d92d9238" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 44947 } }
I20260812 05:24:14.220660 30420 ts_tablet_manager.cc:1428] T bb7ac396618d42f4a0fa1d9ba52e7b6d P 80568930ffe64a73a23282e698c84509: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260812 05:24:14.220703 29934 heartbeater.cc:503] Master 127.31.100.62:40925 was elected leader, sending a full tablet report...
W20260812 05:24:14.304872 30166 tablet.cc:2369] T bb7ac396618d42f4a0fa1d9ba52e7b6d P 69da380f117345049a555d67d92d9238: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 05:24:14.420017 29936 tablet.cc:2369] T bb7ac396618d42f4a0fa1d9ba52e7b6d P 80568930ffe64a73a23282e698c84509: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 05:24:14.433182 30393 tablet.cc:2369] T bb7ac396618d42f4a0fa1d9ba52e7b6d P 9271647c0cc4457db2e58ea0cc2efdb9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 05:24:14.531971 30431 raft_consensus.cc:482] T bb7ac396618d42f4a0fa1d9ba52e7b6d P 9271647c0cc4457db2e58ea0cc2efdb9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 05:24:14.532110 30431 raft_consensus.cc:504] T bb7ac396618d42f4a0fa1d9ba52e7b6d P 9271647c0cc4457db2e58ea0cc2efdb9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9271647c0cc4457db2e58ea0cc2efdb9" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 32799 } } peers { permanent_uuid: "80568930ffe64a73a23282e698c84509" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 41313 } } peers { permanent_uuid: "69da380f117345049a555d67d92d9238" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 44947 } }
I20260812 05:24:14.532473 30431 leader_election.cc:288] T bb7ac396618d42f4a0fa1d9ba52e7b6d P 9271647c0cc4457db2e58ea0cc2efdb9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 80568930ffe64a73a23282e698c84509 (127.31.100.1:41313), 69da380f117345049a555d67d92d9238 (127.31.100.2:44947)
I20260812 05:24:14.535207 30394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:24:14.532643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.1:41313 (local address 127.31.100.3:38351)
0812 05:24:14.532688 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0812 05:24:14.532700 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0812 05:24:14.532748 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:24:14.532893 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:24:14.532894 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:24:14.532896 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 05:24:14.532972 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:24:14.532974 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:24:14.533675 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:24:14.533676 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:24:14.534460 (+ 784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:24:14.534462 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:24:14.534976 (+ 514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:24:14.534977 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:24:14.535043 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:24:14.535047 (+ 4us) client_negotiation.cc:769] Sending connection context
0812 05:24:14.535093 (+ 46us) client_negotiation.cc:240] Negotiation successful
0812 05:24:14.535148 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23,"mutex_wait_us":1}
I20260812 05:24:14.535220 30414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:24:14.532711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.100.3:38351 (local address 127.31.100.1:41313)
0812 05:24:14.532758 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0812 05:24:14.532761 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 05:24:14.532772 (+ 11us) server_negotiation.cc:407] Connection header received
0812 05:24:14.532806 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:24:14.532807 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:24:14.532818 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:24:14.532875 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 05:24:14.533050 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:24:14.533605 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:24:14.534544 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:24:14.534905 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:24:14.534956 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:24:14.534963 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0812 05:24:14.535113 (+ 150us) server_negotiation.cc:299] Negotiation successful
0812 05:24:14.535148 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260812 05:24:14.535326 30434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:24:14.532650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.2:44947 (local address 127.31.100.3:56393)
0812 05:24:14.532869 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0812 05:24:14.532879 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0812 05:24:14.532934 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:24:14.533057 (+ 123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:24:14.533058 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:24:14.533059 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 05:24:14.533123 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:24:14.533125 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:24:14.533807 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:24:14.533808 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:24:14.534566 (+ 758us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:24:14.534568 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:24:14.535088 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:24:14.535089 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:24:14.535163 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:24:14.535167 (+ 4us) client_negotiation.cc:769] Sending connection context
0812 05:24:14.535217 (+ 50us) client_negotiation.cc:240] Negotiation successful
0812 05:24:14.535254 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":198,"mutex_wait_us":59,"thread_start_us":96,"threads_started":1}
I20260812 05:24:14.535341 30417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:24:14.532738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.100.3:56393 (local address 127.31.100.2:44947)
0812 05:24:14.532774 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0812 05:24:14.532776 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 05:24:14.532939 (+ 163us) server_negotiation.cc:407] Connection header received
0812 05:24:14.532974 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:24:14.532975 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:24:14.532995 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:24:14.533050 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 05:24:14.533183 (+ 133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:24:14.533734 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:24:14.534649 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:24:14.535007 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:24:14.535076 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:24:14.535084 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0812 05:24:14.535235 (+ 151us) server_negotiation.cc:299] Negotiation successful
0812 05:24:14.535275 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260812 05:24:14.535712 30088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb7ac396618d42f4a0fa1d9ba52e7b6d" candidate_uuid: "9271647c0cc4457db2e58ea0cc2efdb9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69da380f117345049a555d67d92d9238" is_pre_election: true
I20260812 05:24:14.535706 29859 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb7ac396618d42f4a0fa1d9ba52e7b6d" candidate_uuid: "9271647c0cc4457db2e58ea0cc2efdb9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80568930ffe64a73a23282e698c84509" is_pre_election: true
I20260812 05:24:14.535866 29859 raft_consensus.cc:2454] T bb7ac396618d42f4a0fa1d9ba52e7b6d P 80568930ffe64a73a23282e698c84509 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9271647c0cc4457db2e58ea0cc2efdb9 in term 0.
I20260812 05:24:14.535866 30088 raft_consensus.cc:2454] T bb7ac396618d42f4a0fa1d9ba52e7b6d P 69da380f117345049a555d67d92d9238 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9271647c0cc4457db2e58ea0cc2efdb9 in term 0.
I20260812 05:24:14.536083 30197 leader_election.cc:302] T bb7ac396618d42f4a0fa1d9ba52e7b6d P 9271647c0cc4457db2e58ea0cc2efdb9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 69da380f117345049a555d67d92d9238, 9271647c0cc4457db2e58ea0cc2efdb9; no voters:
I20260812 05:24:14.536227 30431 raft_consensus.cc:2781] T bb7ac396618d42f4a0fa1d9ba52e7b6d P 9271647c0cc4457db2e58ea0cc2efdb9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 05:24:14.536294 30431 raft_consensus.cc:482] T bb7ac396618d42f4a0fa1d9ba52e7b6d P 9271647c0cc4457db2e58ea0cc2efdb9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 05:24:14.536311 30431 raft_consensus.cc:3037] T bb7ac396618d42f4a0fa1d9ba52e7b6d P 9271647c0cc4457db2e58ea0cc2efdb9 [term 0 FOLLOWER]: Advancing to term 1
I20260812 05:24:14.537079 30431 raft_consensus.cc:504] T bb7ac396618d42f4a0fa1d9ba52e7b6d P 9271647c0cc4457db2e58ea0cc2efdb9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9271647c0cc4457db2e58ea0cc2efdb9" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 32799 } } peers { permanent_uuid: "80568930ffe64a73a23282e698c84509" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 41313 } } peers { permanent_uuid: "69da380f117345049a555d67d92d9238" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 44947 } }
I20260812 05:24:14.537204 30431 leader_election.cc:288] T bb7ac396618d42f4a0fa1d9ba52e7b6d P 9271647c0cc4457db2e58ea0cc2efdb9 [CANDIDATE]: Term 1 election: Requested vote from peers 80568930ffe64a73a23282e698c84509 (127.31.100.1:41313), 69da380f117345049a555d67d92d9238 (127.31.100.2:44947)
I20260812 05:24:14.537344 30088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb7ac396618d42f4a0fa1d9ba52e7b6d" candidate_uuid: "9271647c0cc4457db2e58ea0cc2efdb9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69da380f117345049a555d67d92d9238"
I20260812 05:24:14.537343 29859 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb7ac396618d42f4a0fa1d9ba52e7b6d" candidate_uuid: "9271647c0cc4457db2e58ea0cc2efdb9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80568930ffe64a73a23282e698c84509"
I20260812 05:24:14.537420 30088 raft_consensus.cc:3037] T bb7ac396618d42f4a0fa1d9ba52e7b6d P 69da380f117345049a555d67d92d9238 [term 0 FOLLOWER]: Advancing to term 1
I20260812 05:24:14.537423 29859 raft_consensus.cc:3037] T bb7ac396618d42f4a0fa1d9ba52e7b6d P 80568930ffe64a73a23282e698c84509 [term 0 FOLLOWER]: Advancing to term 1
I20260812 05:24:14.538152 30088 raft_consensus.cc:2454] T bb7ac396618d42f4a0fa1d9ba52e7b6d P 69da380f117345049a555d67d92d9238 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9271647c0cc4457db2e58ea0cc2efdb9 in term 1.
I20260812 05:24:14.538226 29859 raft_consensus.cc:2454] T bb7ac396618d42f4a0fa1d9ba52e7b6d P 80568930ffe64a73a23282e698c84509 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9271647c0cc4457db2e58ea0cc2efdb9 in term 1.
I20260812 05:24:14.538314 30197 leader_election.cc:302] T bb7ac396618d42f4a0fa1d9ba52e7b6d P 9271647c0cc4457db2e58ea0cc2efdb9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 69da380f117345049a555d67d92d9238, 9271647c0cc4457db2e58ea0cc2efdb9; no voters:
I20260812 05:24:14.538410 30431 raft_consensus.cc:2781] T bb7ac396618d42f4a0fa1d9ba52e7b6d P 9271647c0cc4457db2e58ea0cc2efdb9 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 05:24:14.538552 30431 raft_consensus.cc:686] T bb7ac396618d42f4a0fa1d9ba52e7b6d P 9271647c0cc4457db2e58ea0cc2efdb9 [term 1 LEADER]: Becoming Leader. State: Replica: 9271647c0cc4457db2e58ea0cc2efdb9, State: Running, Role: LEADER
I20260812 05:24:14.538688 30431 consensus_queue.cc:237] T bb7ac396618d42f4a0fa1d9ba52e7b6d P 9271647c0cc4457db2e58ea0cc2efdb9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9271647c0cc4457db2e58ea0cc2efdb9" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 32799 } } peers { permanent_uuid: "80568930ffe64a73a23282e698c84509" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 41313 } } peers { permanent_uuid: "69da380f117345049a555d67d92d9238" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 44947 } }
I20260812 05:24:14.539479 29621 catalog_manager.cc:5495] T bb7ac396618d42f4a0fa1d9ba52e7b6d P 9271647c0cc4457db2e58ea0cc2efdb9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9271647c0cc4457db2e58ea0cc2efdb9 (127.31.100.3). New cstate: current_term: 1 leader_uuid: "9271647c0cc4457db2e58ea0cc2efdb9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9271647c0cc4457db2e58ea0cc2efdb9" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 32799 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "80568930ffe64a73a23282e698c84509" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 41313 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "69da380f117345049a555d67d92d9238" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 44947 } health_report { overall_health: UNKNOWN } } }
I20260812 05:24:14.564111 30415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:24:14.561943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39032 (local address 127.31.100.3:32799)
0812 05:24:14.561994 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0812 05:24:14.561997 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 05:24:14.562005 (+ 8us) server_negotiation.cc:407] Connection header received
0812 05:24:14.562062 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:24:14.562063 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:24:14.562081 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:24:14.562141 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 05:24:14.562254 (+ 113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:24:14.562708 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:24:14.563421 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:24:14.563609 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:24:14.563663 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:24:14.563805 (+ 142us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 05:24:14.563900 (+ 95us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 05:24:14.563968 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0812 05:24:14.564004 (+ 36us) server_negotiation.cc:299] Negotiation successful
0812 05:24:14.564021 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260812 05:24:14.565750 30088 raft_consensus.cc:1261] T bb7ac396618d42f4a0fa1d9ba52e7b6d P 69da380f117345049a555d67d92d9238 [term 1 FOLLOWER]: Refusing update from remote peer 9271647c0cc4457db2e58ea0cc2efdb9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 05:24:14.565754 29859 raft_consensus.cc:1261] T bb7ac396618d42f4a0fa1d9ba52e7b6d P 80568930ffe64a73a23282e698c84509 [term 1 FOLLOWER]: Refusing update from remote peer 9271647c0cc4457db2e58ea0cc2efdb9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 05:24:14.566061 30435 consensus_queue.cc:1035] T bb7ac396618d42f4a0fa1d9ba52e7b6d P 9271647c0cc4457db2e58ea0cc2efdb9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "80568930ffe64a73a23282e698c84509" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 41313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 05:24:14.566187 30431 consensus_queue.cc:1035] T bb7ac396618d42f4a0fa1d9ba52e7b6d P 9271647c0cc4457db2e58ea0cc2efdb9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69da380f117345049a555d67d92d9238" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 44947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 05:24:14.568799 30442 mvcc.cc:205] Tried to move back new op lower bound from 7317554194698911744 to 7317554194590978048. Current Snapshot: MvccSnapshot[applied={T|T < 7317554194698911744}]
I20260812 05:24:14.570643 29621 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41816:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 05:24:14.570739 29621 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41816:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 05:24:14.571434 29621 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 052e677d293e4c7cb4487efcde0de98a: Sending DeleteTablet for 3 replicas of tablet bb7ac396618d42f4a0fa1d9ba52e7b6d
I20260812 05:24:14.571839 29824 tablet_service.cc:1514] Processing DeleteTablet for tablet bb7ac396618d42f4a0fa1d9ba52e7b6d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 05:24:14 UTC) from {username='slave'} at 127.0.0.1:59252
I20260812 05:24:14.571918 32144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0waFvT/build/release/bin/kudu with pid 29723
I20260812 05:24:14.572371 30277 tablet_service.cc:1514] Processing DeleteTablet for tablet bb7ac396618d42f4a0fa1d9ba52e7b6d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 05:24:14 UTC) from {username='slave'} at 127.0.0.1:39020
I20260812 05:24:14.574689 30053 tablet_service.cc:1514] Processing DeleteTablet for tablet bb7ac396618d42f4a0fa1d9ba52e7b6d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 05:24:14 UTC) from {username='slave'} at 127.0.0.1:55300
I20260812 05:24:14.576388 30458 tablet_replica.cc:330] T bb7ac396618d42f4a0fa1d9ba52e7b6d P 69da380f117345049a555d67d92d9238: stopping tablet replica
I20260812 05:24:14.576542 30458 raft_consensus.cc:2229] T bb7ac396618d42f4a0fa1d9ba52e7b6d P 69da380f117345049a555d67d92d9238 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 05:24:14.576673 30458 raft_consensus.cc:2258] T bb7ac396618d42f4a0fa1d9ba52e7b6d P 69da380f117345049a555d67d92d9238 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 05:24:14.584867 30458 ts_tablet_manager.cc:1905] T bb7ac396618d42f4a0fa1d9ba52e7b6d P 69da380f117345049a555d67d92d9238: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 05:24:14.586287 30458 ts_tablet_manager.cc:1918] T bb7ac396618d42f4a0fa1d9ba52e7b6d P 69da380f117345049a555d67d92d9238: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 05:24:14.586356 30458 log.cc:1196] T bb7ac396618d42f4a0fa1d9ba52e7b6d P 69da380f117345049a555d67d92d9238: Deleting WAL directory at /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786512017613366-32144-0/minicluster-data/ts-1/wal/wals/bb7ac396618d42f4a0fa1d9ba52e7b6d
I20260812 05:24:14.586583 30458 ts_tablet_manager.cc:1939] T bb7ac396618d42f4a0fa1d9ba52e7b6d P 69da380f117345049a555d67d92d9238: Deleting consensus metadata
I20260812 05:24:14.586974 29601 catalog_manager.cc:4841] TS 69da380f117345049a555d67d92d9238 (127.31.100.2:44947): tablet bb7ac396618d42f4a0fa1d9ba52e7b6d (table test-table [id=a9f4e6fd99cc471397f7751af8c483f7]) successfully deleted
I20260812 05:24:14.593082 30457 tablet_replica.cc:330] T bb7ac396618d42f4a0fa1d9ba52e7b6d P 9271647c0cc4457db2e58ea0cc2efdb9: stopping tablet replica
I20260812 05:24:14.593206 30457 raft_consensus.cc:2229] T bb7ac396618d42f4a0fa1d9ba52e7b6d P 9271647c0cc4457db2e58ea0cc2efdb9 [term 1 LEADER]: Raft consensus shutting down.
I20260812 05:24:14.593309 30457 raft_consensus.cc:2258] T bb7ac396618d42f4a0fa1d9ba52e7b6d P 9271647c0cc4457db2e58ea0cc2efdb9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 05:24:14.593851 30457 ts_tablet_manager.cc:1905] T bb7ac396618d42f4a0fa1d9ba52e7b6d P 9271647c0cc4457db2e58ea0cc2efdb9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 05:24:14.595429 30457 ts_tablet_manager.cc:1918] T bb7ac396618d42f4a0fa1d9ba52e7b6d P 9271647c0cc4457db2e58ea0cc2efdb9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 05:24:14.595505 30457 log.cc:1196] T bb7ac396618d42f4a0fa1d9ba52e7b6d P 9271647c0cc4457db2e58ea0cc2efdb9: Deleting WAL directory at /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786512017613366-32144-0/minicluster-data/ts-2/wal/wals/bb7ac396618d42f4a0fa1d9ba52e7b6d
I20260812 05:24:14.595769 30457 ts_tablet_manager.cc:1939] T bb7ac396618d42f4a0fa1d9ba52e7b6d P 9271647c0cc4457db2e58ea0cc2efdb9: Deleting consensus metadata
I20260812 05:24:14.596167 29601 catalog_manager.cc:4841] TS 9271647c0cc4457db2e58ea0cc2efdb9 (127.31.100.3:32799): tablet bb7ac396618d42f4a0fa1d9ba52e7b6d (table test-table [id=a9f4e6fd99cc471397f7751af8c483f7]) successfully deleted
W20260812 05:24:14.597057 29597 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.31.100.1:41313 (error 108)
I20260812 05:24:14.597546 30413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:24:14.597398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.1:41313 (local address 127.0.0.1:59262)
0812 05:24:14.597466 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0812 05:24:14.597511 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.100.1:41313: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260812 05:24:14.597676 32144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0waFvT/build/release/bin/kudu with pid 29938
W20260812 05:24:14.597674 29597 catalog_manager.cc:4584] TS 80568930ffe64a73a23282e698c84509 (127.31.100.1:41313): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet bb7ac396618d42f4a0fa1d9ba52e7b6d: Network error: Client connection negotiation failed: client connection to 127.31.100.1:41313: connect: Connection refused (error 111)
I20260812 05:24:14.611935 32144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0waFvT/build/release/bin/kudu with pid 30171
I20260812 05:24:14.631405 32144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0waFvT/build/release/bin/kudu with pid 29567
2026-08-12T05:24:14Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (1007 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786512017613366-32144-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 05:24:14 dist-test-slave-42z9 krb5kdc[30462](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 05:24:14 dist-test-slave-42z9 krb5kdc[30462](info): set up 2 sockets
Aug 12 05:24:14 dist-test-slave-42z9 krb5kdc[30462](info): commencing operation
krb5kdc: starting...
W20260812 05:24:15.797857 32144 mini_kdc.cc:121] Time spent starting KDC: real 1.143s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786512017613366-32144-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786512017613366-32144-0/krb5kdc/test-user.keytab.
Aug 12 05:24:15 dist-test-slave-42z9 krb5kdc[30462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512255, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T05:24:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T05:24:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.100.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.100.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.100.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.62.keytab.
Entry for principal kudu/127.31.100.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.62.keytab.
I20260812 05:24:15.907797 32144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0waFvT/build/release/bin/kudu
/tmp/dist-test-task0waFvT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786512017613366-32144-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786512017613366-32144-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786512017613366-32144-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786512017613366-32144-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.100.62:40857
--webserver_interface=127.31.100.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.62.keytab
--principal=kudu/127.31.100.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.100.20:35597
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.100.62:40857
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786512017613366-32144-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786512017613366-32144-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786512017613366-32144-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 05:24:15.986979 30485 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 05:24:15.987177 30485 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 05:24:15.987207 30485 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 05:24:15.988325 30485 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 05:24:15.988366 30485 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 05:24:15.988384 30485 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 05:24:15.988400 30485 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 05:24:15.988415 30485 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 05:24:15.989926 30485 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.100.20:35597
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786512017613366-32144-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786512017613366-32144-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.100.62:40857
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.62.keytab
--principal=kudu/127.31.100.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.100.62:40857
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786512017613366-32144-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.100.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30485
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786512017613366-32144-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 04:20:34 UTC on ccycloud.quasar-phqdqp.root.comops.site
build id 4581871
I20260812 05:24:15.990175 30485 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 05:24:15.990370 30485 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 05:24:15.994905 30490 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 05:24:15.994896 30492 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 05:24:15.995020 30485 server_base.cc:1034] running on GCE node
W20260812 05:24:15.994899 30495 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 05:24:15.995332 30485 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 05:24:15.995612 30485 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 05:24:15.996737 30485 hybrid_clock.cc:648] HybridClock initialized: now 1786512255996725 us; error 34 us; skew 500 ppm
Aug 12 05:24:16 dist-test-slave-42z9 krb5kdc[30462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512256, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.100.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 05:24:16.003750 30485 init.cc:377] Logged in from keytab as kudu/127.31.100.62@KRBTEST.COM (short username kudu)
I20260812 05:24:16.004158 30485 webserver.cc:460] Webserver started at http://127.31.100.62:37819/ using document root <none> and password file <none>
I20260812 05:24:16.004364 30485 fs_manager.cc:359] Metadata directory not provided
I20260812 05:24:16.004410 30485 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 05:24:16.004540 30485 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 05:24:16.005589 30485 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786512017613366-32144-0/minicluster-data/master-0/data/instance:
uuid: "7a40e95cce1d460485805b5d07145b95"
format_stamp: "Formatted at 2026-08-12 05:24:16 on dist-test-slave-42z9"
server_key: "01a70b6d8e6a61fbb39afc9dcd77ea8d"
server_key_iv: "e728889e1cec1fbaca890eeac7e2c6f9"
server_key_version: "encryptionkey@0"
I20260812 05:24:16.005905 30485 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786512017613366-32144-0/minicluster-data/master-0/wal/instance:
uuid: "7a40e95cce1d460485805b5d07145b95"
format_stamp: "Formatted at 2026-08-12 05:24:16 on dist-test-slave-42z9"
server_key: "01a70b6d8e6a61fbb39afc9dcd77ea8d"
server_key_iv: "e728889e1cec1fbaca890eeac7e2c6f9"
server_key_version: "encryptionkey@0"
I20260812 05:24:16.007093 30485 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260812 05:24:16.007723 30505 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 05:24:16.007875 30485 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260812 05:24:16.007961 30485 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786512017613366-32144-0/minicluster-data/master-0/data,/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786512017613366-32144-0/minicluster-data/master-0/wal
uuid: "7a40e95cce1d460485805b5d07145b95"
format_stamp: "Formatted at 2026-08-12 05:24:16 on dist-test-slave-42z9"
server_key: "01a70b6d8e6a61fbb39afc9dcd77ea8d"
server_key_iv: "e728889e1cec1fbaca890eeac7e2c6f9"
server_key_version: "encryptionkey@0"
I20260812 05:24:16.008016 30485 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786512017613366-32144-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786512017613366-32144-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786512017613366-32144-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 05:24:16.039316 30485 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 05:24:16.041085 30485 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 05:24:16.041194 30485 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 05:24:16.044991 30485 rpc_server.cc:307] RPC server started. Bound to: 127.31.100.62:40857
I20260812 05:24:16.045030 30599 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.100.62:40857 every 8 connection(s)
I20260812 05:24:16.045338 30485 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786512017613366-32144-0/minicluster-data/master-0/data/info.pb
I20260812 05:24:16.046200 30601 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 05:24:16.048434 30601 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7a40e95cce1d460485805b5d07145b95: Bootstrap starting.
I20260812 05:24:16.048982 30601 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7a40e95cce1d460485805b5d07145b95: Neither blocks nor log segments found. Creating new log.
I20260812 05:24:16.049263 30601 log.cc:824] T 00000000000000000000000000000000 P 7a40e95cce1d460485805b5d07145b95: Log is configured to *not* fsync() on all Append() calls
I20260812 05:24:16.049935 30601 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7a40e95cce1d460485805b5d07145b95: No bootstrap required, opened a new log
I20260812 05:24:16.051486 30601 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7a40e95cce1d460485805b5d07145b95 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a40e95cce1d460485805b5d07145b95" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 40857 } }
I20260812 05:24:16.051620 30601 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7a40e95cce1d460485805b5d07145b95 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 05:24:16.051652 30601 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7a40e95cce1d460485805b5d07145b95 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a40e95cce1d460485805b5d07145b95, State: Initialized, Role: FOLLOWER
I20260812 05:24:16.051793 30601 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7a40e95cce1d460485805b5d07145b95 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a40e95cce1d460485805b5d07145b95" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 40857 } }
I20260812 05:24:16.051888 30601 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7a40e95cce1d460485805b5d07145b95 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 05:24:16.051928 30601 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7a40e95cce1d460485805b5d07145b95 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 05:24:16.051975 30601 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7a40e95cce1d460485805b5d07145b95 [term 0 FOLLOWER]: Advancing to term 1
I20260812 05:24:16.052069 32144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0waFvT/build/release/bin/kudu as pid 30485
I20260812 05:24:16.052186 32144 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786512017613366-32144-0/minicluster-data/master-0/wal/instance
I20260812 05:24:16.052362 32144 external_mini_cluster.cc:1428] Setting key 2b8d2147a4404bd199b0d6b7e75dc0a7
I20260812 05:24:16.052569 30601 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7a40e95cce1d460485805b5d07145b95 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a40e95cce1d460485805b5d07145b95" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 40857 } }
I20260812 05:24:16.052676 30601 leader_election.cc:302] T 00000000000000000000000000000000 P 7a40e95cce1d460485805b5d07145b95 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7a40e95cce1d460485805b5d07145b95; no voters:
I20260812 05:24:16.052845 30601 leader_election.cc:288] T 00000000000000000000000000000000 P 7a40e95cce1d460485805b5d07145b95 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 05:24:16.052933 30607 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7a40e95cce1d460485805b5d07145b95 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 05:24:16.053081 30607 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7a40e95cce1d460485805b5d07145b95 [term 1 LEADER]: Becoming Leader. State: Replica: 7a40e95cce1d460485805b5d07145b95, State: Running, Role: LEADER
I20260812 05:24:16.053190 30601 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7a40e95cce1d460485805b5d07145b95 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 05:24:16.053192 30607 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7a40e95cce1d460485805b5d07145b95 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a40e95cce1d460485805b5d07145b95" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 40857 } }
I20260812 05:24:16.055106 30609 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a40e95cce1d460485805b5d07145b95 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7a40e95cce1d460485805b5d07145b95" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a40e95cce1d460485805b5d07145b95" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 40857 } } }
I20260812 05:24:16.055204 30609 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a40e95cce1d460485805b5d07145b95 [sys.catalog]: This master's current role is: LEADER
I20260812 05:24:16.055114 30610 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a40e95cce1d460485805b5d07145b95 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7a40e95cce1d460485805b5d07145b95. Latest consensus state: current_term: 1 leader_uuid: "7a40e95cce1d460485805b5d07145b95" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a40e95cce1d460485805b5d07145b95" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 40857 } } }
I20260812 05:24:16.055341 30610 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a40e95cce1d460485805b5d07145b95 [sys.catalog]: This master's current role is: LEADER
W20260812 05:24:16.055959 30632 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7a40e95cce1d460485805b5d07145b95: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 05:24:16.056740 30632 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 05:24:16.056818 30629 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 05:24:16.056986 30629 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 05:24:16.060166 30629 catalog_manager.cc:1347] Generated new cluster ID: 77c231ae45f1429a91b5c781a4b94c7e
I20260812 05:24:16.060221 30629 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 12 05:24:16 dist-test-slave-42z9 krb5kdc[30462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512255, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.100.62@KRBTEST.COM
I20260812 05:24:16.064695 30608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:24:16.052782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55566 (local address 127.31.100.62:40857)
0812 05:24:16.052934 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0812 05:24:16.052944 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 05:24:16.052981 (+ 37us) server_negotiation.cc:407] Connection header received
0812 05:24:16.053130 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:24:16.053138 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:24:16.053194 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:24:16.053357 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:24:16.054208 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:24:16.055537 (+ 1329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:24:16.056489 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:24:16.057066 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:24:16.057173 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:24:16.061706 (+ 4533us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 05:24:16.061710 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 05:24:16.061710 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 05:24:16.061723 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0812 05:24:16.063297 (+ 1574us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:24:16.063622 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:24:16.063625 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:24:16.063625 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:24:16.063723 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:24:16.063904 (+ 181us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:24:16.063905 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:24:16.063905 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:24:16.064139 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 05:24:16.064288 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0812 05:24:16.064383 (+ 95us) server_negotiation.cc:299] Negotiation successful
0812 05:24:16.064516 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102,"thread_start_us":67,"threads_started":1}
I20260812 05:24:16.069914 30629 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 05:24:16.070441 30629 catalog_manager.cc:1504] Loading token signing keys...
I20260812 05:24:16.075748 30629 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7a40e95cce1d460485805b5d07145b95: Generated new TSK 0
I20260812 05:24:16.075911 30629 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.31.100.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.100.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.100.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.1.keytab.
Entry for principal kudu/127.31.100.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.1.keytab.
I20260812 05:24:16.107968 32144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0waFvT/build/release/bin/kudu
/tmp/dist-test-task0waFvT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786512017613366-32144-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786512017613366-32144-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786512017613366-32144-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786512017613366-32144-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.100.1:0
--local_ip_for_outbound_sockets=127.31.100.1
--webserver_interface=127.31.100.1
--webserver_port=0
--tserver_master_addrs=127.31.100.62:40857
--keytab_file=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.1.keytab
--principal=kudu/127.31.100.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.100.20:35597
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786512017613366-32144-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786512017613366-32144-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786512017613366-32144-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 05:24:16.208305 30640 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 05:24:16.208501 30640 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 05:24:16.208523 30640 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 05:24:16.209659 30640 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 05:24:16.209697 30640 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 05:24:16.209728 30640 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.100.1
I20260812 05:24:16.211233 30640 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.100.20:35597
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786512017613366-32144-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786512017613366-32144-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.1.keytab
--principal=kudu/127.31.100.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.100.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786512017613366-32144-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.100.1
--webserver_port=0
--tserver_master_addrs=127.31.100.62:40857
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30640
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.100.1
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786512017613366-32144-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 04:20:34 UTC on ccycloud.quasar-phqdqp.root.comops.site
build id 4581871
I20260812 05:24:16.211442 30640 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 05:24:16.211687 30640 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 05:24:16.216399 30651 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 05:24:16.216399 30646 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 05:24:16.216497 30648 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 05:24:16.216548 30640 server_base.cc:1034] running on GCE node
I20260812 05:24:16.216815 30640 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 05:24:16.217049 30640 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 05:24:16.218186 30640 hybrid_clock.cc:648] HybridClock initialized: now 1786512256218173 us; error 29 us; skew 500 ppm
Aug 12 05:24:16 dist-test-slave-42z9 krb5kdc[30462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512256, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.100.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 05:24:16.220489 30640 init.cc:377] Logged in from keytab as kudu/127.31.100.1@KRBTEST.COM (short username kudu)
I20260812 05:24:16.220846 30640 webserver.cc:460] Webserver started at http://127.31.100.1:36991/ using document root <none> and password file <none>
I20260812 05:24:16.221015 30640 fs_manager.cc:359] Metadata directory not provided
I20260812 05:24:16.221055 30640 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 05:24:16.221140 30640 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 05:24:16.222095 30640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786512017613366-32144-0/minicluster-data/ts-0/data/instance:
uuid: "0d6f22f2631e4e6aadad889de48ea392"
format_stamp: "Formatted at 2026-08-12 05:24:16 on dist-test-slave-42z9"
server_key: "b54bf6d9a1159338a7967f30110d39f1"
server_key_iv: "7588ba60e8d29f12c438ca2ecefc5179"
server_key_version: "encryptionkey@0"
I20260812 05:24:16.222373 30640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786512017613366-32144-0/minicluster-data/ts-0/wal/instance:
uuid: "0d6f22f2631e4e6aadad889de48ea392"
format_stamp: "Formatted at 2026-08-12 05:24:16 on dist-test-slave-42z9"
server_key: "b54bf6d9a1159338a7967f30110d39f1"
server_key_iv: "7588ba60e8d29f12c438ca2ecefc5179"
server_key_version: "encryptionkey@0"
I20260812 05:24:16.223536 30640 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260812 05:24:16.224154 30662 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 05:24:16.224289 30640 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260812 05:24:16.224366 30640 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786512017613366-32144-0/minicluster-data/ts-0/data,/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786512017613366-32144-0/minicluster-data/ts-0/wal
uuid: "0d6f22f2631e4e6aadad889de48ea392"
format_stamp: "Formatted at 2026-08-12 05:24:16 on dist-test-slave-42z9"
server_key: "b54bf6d9a1159338a7967f30110d39f1"
server_key_iv: "7588ba60e8d29f12c438ca2ecefc5179"
server_key_version: "encryptionkey@0"
I20260812 05:24:16.224427 30640 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786512017613366-32144-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786512017613366-32144-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786512017613366-32144-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 05:24:16.237504 30640 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 05:24:16.239468 30640 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 05:24:16.239617 30640 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 05:24:16.239837 30640 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 05:24:16.240129 30640 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 05:24:16.240162 30640 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 05:24:16.240182 30640 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 05:24:16.240204 30640 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 05:24:16.245641 30640 rpc_server.cc:307] RPC server started. Bound to: 127.31.100.1:39749
I20260812 05:24:16.245700 30861 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.100.1:39749 every 8 connection(s)
I20260812 05:24:16.245975 30640 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786512017613366-32144-0/minicluster-data/ts-0/data/info.pb
Aug 12 05:24:16 dist-test-slave-42z9 krb5kdc[30462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512256, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.100.1@KRBTEST.COM for kudu/127.31.100.62@KRBTEST.COM
I20260812 05:24:16.253142 32144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0waFvT/build/release/bin/kudu as pid 30640
I20260812 05:24:16.253268 32144 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786512017613366-32144-0/minicluster-data/ts-0/wal/instance
I20260812 05:24:16.253513 32144 external_mini_cluster.cc:1428] Setting key 9f61dcf38b3fb9128dbc551a3b2713db
I20260812 05:24:16.254349 30608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:24:16.246591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.100.1:37967 (local address 127.31.100.62:40857)
0812 05:24:16.246673 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0812 05:24:16.246675 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 05:24:16.246816 (+ 141us) server_negotiation.cc:407] Connection header received
0812 05:24:16.247252 (+ 436us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:24:16.247253 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:24:16.247262 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:24:16.247306 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:24:16.248156 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:24:16.248648 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:24:16.249620 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:24:16.249820 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:24:16.249875 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:24:16.251723 (+ 1848us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 05:24:16.251725 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 05:24:16.251725 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 05:24:16.251731 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0812 05:24:16.252873 (+ 1142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:24:16.253352 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:24:16.253354 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:24:16.253354 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:24:16.253417 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:24:16.253714 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:24:16.253716 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:24:16.253716 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:24:16.253857 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 05:24:16.253921 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0812 05:24:16.254195 (+ 274us) server_negotiation.cc:299] Negotiation successful
0812 05:24:16.254286 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260812 05:24:16.254516 30864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:24:16.246556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.62:40857 (local address 127.31.100.1:37967)
0812 05:24:16.246709 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0812 05:24:16.246742 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0812 05:24:16.247175 (+ 433us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:24:16.247364 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:24:16.247367 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:24:16.247901 (+ 534us) client_negotiation.cc:189] Negotiated authn=SASL
0812 05:24:16.248087 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:24:16.248092 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:24:16.248724 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:24:16.248725 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:24:16.249539 (+ 814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:24:16.249541 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:24:16.249874 (+ 333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:24:16.249875 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:24:16.249962 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:24:16.250316 (+ 354us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 05:24:16.250323 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0812 05:24:16.251643 (+ 1320us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 05:24:16.252941 (+ 1298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 05:24:16.252949 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 05:24:16.252952 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 05:24:16.253269 (+ 317us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 05:24:16.253514 (+ 245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 05:24:16.253515 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 05:24:16.253515 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 05:24:16.253650 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 05:24:16.253911 (+ 261us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 05:24:16.253918 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 05:24:16.254048 (+ 130us) client_negotiation.cc:769] Sending connection context
0812 05:24:16.254147 (+ 99us) client_negotiation.cc:240] Negotiation successful
0812 05:24:16.254308 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":78,"threads_started":1}
I20260812 05:24:16.254976 30862 heartbeater.cc:347] Connected to a master server at 127.31.100.62:40857
I20260812 05:24:16.255074 30862 heartbeater.cc:464] Registering TS with master...
I20260812 05:24:16.255257 30862 heartbeater.cc:511] Master 127.31.100.62:40857 requested a full tablet report, sending...
I20260812 05:24:16.255781 30537 ts_manager.cc:194] Registered new tserver with Master: 0d6f22f2631e4e6aadad889de48ea392 (127.31.100.1:39749)
I20260812 05:24:16.256399 30537 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.100.1@KRBTEST.COM'} at 127.31.100.1:37967
WARNING: no policy specified for kudu/127.31.100.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.100.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.100.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.2.keytab.
Entry for principal kudu/127.31.100.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.2.keytab.
I20260812 05:24:16.277593 32144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0waFvT/build/release/bin/kudu
/tmp/dist-test-task0waFvT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786512017613366-32144-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786512017613366-32144-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786512017613366-32144-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786512017613366-32144-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.100.2:0
--local_ip_for_outbound_sockets=127.31.100.2
--webserver_interface=127.31.100.2
--webserver_port=0
--tserver_master_addrs=127.31.100.62:40857
--keytab_file=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.2.keytab
--principal=kudu/127.31.100.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.100.20:35597
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786512017613366-32144-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786512017613366-32144-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786512017613366-32144-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 05:24:16.356413 30869 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 05:24:16.356598 30869 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 05:24:16.356622 30869 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 05:24:16.357738 30869 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 05:24:16.357776 30869 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 05:24:16.357800 30869 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.100.2
I20260812 05:24:16.359229 30869 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.100.20:35597
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786512017613366-32144-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786512017613366-32144-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.2.keytab
--principal=kudu/127.31.100.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.100.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786512017613366-32144-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.100.2
--webserver_port=0
--tserver_master_addrs=127.31.100.62:40857
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30869
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.100.2
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786512017613366-32144-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 04:20:34 UTC on ccycloud.quasar-phqdqp.root.comops.site
build id 4581871
I20260812 05:24:16.359448 30869 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 05:24:16.359658 30869 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 05:24:16.364318 30877 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 05:24:16.364326 30879 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 05:24:16.364318 30881 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 05:24:16.364445 30869 server_base.cc:1034] running on GCE node
I20260812 05:24:16.364692 30869 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 05:24:16.364925 30869 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 05:24:16.366086 30869 hybrid_clock.cc:648] HybridClock initialized: now 1786512256366073 us; error 28 us; skew 500 ppm
Aug 12 05:24:16 dist-test-slave-42z9 krb5kdc[30462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512256, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.100.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 05:24:16.368391 30869 init.cc:377] Logged in from keytab as kudu/127.31.100.2@KRBTEST.COM (short username kudu)
I20260812 05:24:16.368783 30869 webserver.cc:460] Webserver started at http://127.31.100.2:38739/ using document root <none> and password file <none>
I20260812 05:24:16.368986 30869 fs_manager.cc:359] Metadata directory not provided
I20260812 05:24:16.369035 30869 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 05:24:16.369158 30869 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 05:24:16.370152 30869 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786512017613366-32144-0/minicluster-data/ts-1/data/instance:
uuid: "00f8cc2a396a4a43becb5e9ec2ab21a8"
format_stamp: "Formatted at 2026-08-12 05:24:16 on dist-test-slave-42z9"
server_key: "c9d5b6b72f4a88385c299188256f1f96"
server_key_iv: "7bc3c4b4ced2c7d54c85fde07fd9425f"
server_key_version: "encryptionkey@0"
I20260812 05:24:16.370455 30869 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786512017613366-32144-0/minicluster-data/ts-1/wal/instance:
uuid: "00f8cc2a396a4a43becb5e9ec2ab21a8"
format_stamp: "Formatted at 2026-08-12 05:24:16 on dist-test-slave-42z9"
server_key: "c9d5b6b72f4a88385c299188256f1f96"
server_key_iv: "7bc3c4b4ced2c7d54c85fde07fd9425f"
server_key_version: "encryptionkey@0"
I20260812 05:24:16.371706 30869 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260812 05:24:16.372332 30894 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 05:24:16.372475 30869 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260812 05:24:16.372550 30869 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786512017613366-32144-0/minicluster-data/ts-1/data,/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786512017613366-32144-0/minicluster-data/ts-1/wal
uuid: "00f8cc2a396a4a43becb5e9ec2ab21a8"
format_stamp: "Formatted at 2026-08-12 05:24:16 on dist-test-slave-42z9"
server_key: "c9d5b6b72f4a88385c299188256f1f96"
server_key_iv: "7bc3c4b4ced2c7d54c85fde07fd9425f"
server_key_version: "encryptionkey@0"
I20260812 05:24:16.372613 30869 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786512017613366-32144-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786512017613366-32144-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786512017613366-32144-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 05:24:16.380290 30869 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 05:24:16.381871 30869 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 05:24:16.381987 30869 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 05:24:16.382170 30869 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 05:24:16.382439 30869 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 05:24:16.382470 30869 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 05:24:16.382489 30869 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 05:24:16.382512 30869 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 05:24:16.387689 30869 rpc_server.cc:307] RPC server started. Bound to: 127.31.100.2:41199
I20260812 05:24:16.387751 31082 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.100.2:41199 every 8 connection(s)
I20260812 05:24:16.387989 30869 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786512017613366-32144-0/minicluster-data/ts-1/data/info.pb
I20260812 05:24:16.391834 32144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0waFvT/build/release/bin/kudu as pid 30869
I20260812 05:24:16.391937 32144 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786512017613366-32144-0/minicluster-data/ts-1/wal/instance
I20260812 05:24:16.392127 32144 external_mini_cluster.cc:1428] Setting key e3ff9c9d0560a2127603bba20f4535bc
Aug 12 05:24:16 dist-test-slave-42z9 krb5kdc[30462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512256, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.100.2@KRBTEST.COM for kudu/127.31.100.62@KRBTEST.COM
I20260812 05:24:16.396735 30608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:24:16.388610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.100.2:48777 (local address 127.31.100.62:40857)
0812 05:24:16.388666 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0812 05:24:16.388668 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 05:24:16.388824 (+ 156us) server_negotiation.cc:407] Connection header received
0812 05:24:16.389291 (+ 467us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:24:16.389293 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:24:16.389306 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:24:16.389357 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:24:16.390277 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:24:16.390775 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:24:16.391750 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:24:16.392028 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:24:16.392086 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:24:16.394285 (+ 2199us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 05:24:16.394287 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 05:24:16.394287 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 05:24:16.394295 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0812 05:24:16.395562 (+ 1267us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:24:16.395908 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:24:16.395909 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:24:16.395910 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:24:16.395975 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:24:16.396178 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:24:16.396179 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:24:16.396179 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:24:16.396298 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 05:24:16.396363 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0812 05:24:16.396563 (+ 200us) server_negotiation.cc:299] Negotiation successful
0812 05:24:16.396652 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260812 05:24:16.396781 31090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:24:16.388584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.62:40857 (local address 127.31.100.2:48777)
0812 05:24:16.388718 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0812 05:24:16.388751 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0812 05:24:16.389196 (+ 445us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:24:16.389418 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:24:16.389421 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:24:16.389984 (+ 563us) client_negotiation.cc:189] Negotiated authn=SASL
0812 05:24:16.390174 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:24:16.390179 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:24:16.390858 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:24:16.390859 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:24:16.391658 (+ 799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:24:16.391660 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:24:16.392124 (+ 464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:24:16.392125 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:24:16.392255 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:24:16.392731 (+ 476us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 05:24:16.392740 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0812 05:24:16.394198 (+ 1458us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 05:24:16.395628 (+ 1430us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 05:24:16.395634 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 05:24:16.395636 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 05:24:16.395841 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 05:24:16.396034 (+ 193us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 05:24:16.396034 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 05:24:16.396034 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 05:24:16.396129 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 05:24:16.396353 (+ 224us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 05:24:16.396356 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 05:24:16.396452 (+ 96us) client_negotiation.cc:769] Sending connection context
0812 05:24:16.396521 (+ 69us) client_negotiation.cc:240] Negotiation successful
0812 05:24:16.396636 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":67,"threads_started":1}
I20260812 05:24:16.397203 31083 heartbeater.cc:347] Connected to a master server at 127.31.100.62:40857
I20260812 05:24:16.397296 31083 heartbeater.cc:464] Registering TS with master...
I20260812 05:24:16.397491 31083 heartbeater.cc:511] Master 127.31.100.62:40857 requested a full tablet report, sending...
I20260812 05:24:16.397866 30537 ts_manager.cc:194] Registered new tserver with Master: 00f8cc2a396a4a43becb5e9ec2ab21a8 (127.31.100.2:41199)
I20260812 05:24:16.398227 30537 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.100.2@KRBTEST.COM'} at 127.31.100.2:48777
WARNING: no policy specified for kudu/127.31.100.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.100.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.100.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.3.keytab.
Entry for principal kudu/127.31.100.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.3.keytab.
I20260812 05:24:16.416672 32144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0waFvT/build/release/bin/kudu
/tmp/dist-test-task0waFvT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786512017613366-32144-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786512017613366-32144-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786512017613366-32144-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786512017613366-32144-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.100.3:0
--local_ip_for_outbound_sockets=127.31.100.3
--webserver_interface=127.31.100.3
--webserver_port=0
--tserver_master_addrs=127.31.100.62:40857
--keytab_file=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.3.keytab
--principal=kudu/127.31.100.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.100.20:35597
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786512017613366-32144-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786512017613366-32144-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786512017613366-32144-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 05:24:16.496093 31093 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 05:24:16.496282 31093 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 05:24:16.496304 31093 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 05:24:16.497361 31093 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 05:24:16.497397 31093 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 05:24:16.497422 31093 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.100.3
I20260812 05:24:16.498888 31093 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.100.20:35597
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786512017613366-32144-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786512017613366-32144-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.3.keytab
--principal=kudu/127.31.100.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.100.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786512017613366-32144-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.100.3
--webserver_port=0
--tserver_master_addrs=127.31.100.62:40857
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31093
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.100.3
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786512017613366-32144-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 04:20:34 UTC on ccycloud.quasar-phqdqp.root.comops.site
build id 4581871
I20260812 05:24:16.499074 31093 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 05:24:16.499254 31093 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 05:24:16.503931 31103 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 05:24:16.503935 31105 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 05:24:16.504161 31108 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 05:24:16.504153 31093 server_base.cc:1034] running on GCE node
I20260812 05:24:16.504354 31093 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 05:24:16.504552 31093 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 05:24:16.505695 31093 hybrid_clock.cc:648] HybridClock initialized: now 1786512256505683 us; error 29 us; skew 500 ppm
Aug 12 05:24:16 dist-test-slave-42z9 krb5kdc[30462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512256, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.100.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 05:24:16.507979 31093 init.cc:377] Logged in from keytab as kudu/127.31.100.3@KRBTEST.COM (short username kudu)
I20260812 05:24:16.508364 31093 webserver.cc:460] Webserver started at http://127.31.100.3:39051/ using document root <none> and password file <none>
I20260812 05:24:16.508575 31093 fs_manager.cc:359] Metadata directory not provided
I20260812 05:24:16.508622 31093 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 05:24:16.508746 31093 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 05:24:16.509735 31093 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786512017613366-32144-0/minicluster-data/ts-2/data/instance:
uuid: "8d5863a178ae414f897ea77e4dc693ea"
format_stamp: "Formatted at 2026-08-12 05:24:16 on dist-test-slave-42z9"
server_key: "7e005540d6f3be96c5e69bda8f741c99"
server_key_iv: "dd6998c64b8e5d02ac2b8be0c1dd5fb2"
server_key_version: "encryptionkey@0"
I20260812 05:24:16.510066 31093 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786512017613366-32144-0/minicluster-data/ts-2/wal/instance:
uuid: "8d5863a178ae414f897ea77e4dc693ea"
format_stamp: "Formatted at 2026-08-12 05:24:16 on dist-test-slave-42z9"
server_key: "7e005540d6f3be96c5e69bda8f741c99"
server_key_iv: "dd6998c64b8e5d02ac2b8be0c1dd5fb2"
server_key_version: "encryptionkey@0"
I20260812 05:24:16.511286 31093 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260812 05:24:16.511933 31125 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 05:24:16.512084 31093 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260812 05:24:16.512163 31093 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786512017613366-32144-0/minicluster-data/ts-2/data,/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786512017613366-32144-0/minicluster-data/ts-2/wal
uuid: "8d5863a178ae414f897ea77e4dc693ea"
format_stamp: "Formatted at 2026-08-12 05:24:16 on dist-test-slave-42z9"
server_key: "7e005540d6f3be96c5e69bda8f741c99"
server_key_iv: "dd6998c64b8e5d02ac2b8be0c1dd5fb2"
server_key_version: "encryptionkey@0"
I20260812 05:24:16.512219 31093 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786512017613366-32144-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786512017613366-32144-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786512017613366-32144-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 05:24:16.520767 31093 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 05:24:16.522404 31093 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 05:24:16.522538 31093 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 05:24:16.522727 31093 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 05:24:16.523015 31093 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 05:24:16.523049 31093 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 05:24:16.523077 31093 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 05:24:16.523093 31093 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 05:24:16.528535 31093 rpc_server.cc:307] RPC server started. Bound to: 127.31.100.3:33651
I20260812 05:24:16.528597 31296 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.100.3:33651 every 8 connection(s)
I20260812 05:24:16.528844 31093 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786512017613366-32144-0/minicluster-data/ts-2/data/info.pb
I20260812 05:24:16.531047 32144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0waFvT/build/release/bin/kudu as pid 31093
I20260812 05:24:16.531157 32144 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786512017613366-32144-0/minicluster-data/ts-2/wal/instance
I20260812 05:24:16.531347 32144 external_mini_cluster.cc:1428] Setting key 542a7f6afcd994bcefccb1f0a55e36b3
Aug 12 05:24:16 dist-test-slave-42z9 krb5kdc[30462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512256, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.100.3@KRBTEST.COM for kudu/127.31.100.62@KRBTEST.COM
I20260812 05:24:16.537297 30608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:24:16.529479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.100.3:35607 (local address 127.31.100.62:40857)
0812 05:24:16.529524 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0812 05:24:16.529526 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 05:24:16.529741 (+ 215us) server_negotiation.cc:407] Connection header received
0812 05:24:16.530152 (+ 411us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:24:16.530153 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:24:16.530164 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:24:16.530212 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:24:16.531120 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:24:16.531735 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:24:16.532820 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:24:16.533017 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:24:16.533065 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:24:16.534975 (+ 1910us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 05:24:16.534976 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 05:24:16.534976 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 05:24:16.534982 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0812 05:24:16.536044 (+ 1062us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:24:16.536446 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:24:16.536447 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:24:16.536447 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:24:16.536500 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:24:16.536714 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:24:16.536715 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:24:16.536716 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:24:16.536835 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 05:24:16.536899 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0812 05:24:16.537167 (+ 268us) server_negotiation.cc:299] Negotiation successful
0812 05:24:16.537234 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260812 05:24:16.537407 31300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:24:16.529479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.62:40857 (local address 127.31.100.3:35607)
0812 05:24:16.529624 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0812 05:24:16.529656 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0812 05:24:16.530080 (+ 424us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:24:16.530261 (+ 181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:24:16.530264 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:24:16.530781 (+ 517us) client_negotiation.cc:189] Negotiated authn=SASL
0812 05:24:16.531025 (+ 244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:24:16.531032 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:24:16.531807 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:24:16.531808 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:24:16.532752 (+ 944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:24:16.532754 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:24:16.533074 (+ 320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:24:16.533074 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:24:16.533170 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:24:16.533651 (+ 481us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 05:24:16.533659 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0812 05:24:16.534904 (+ 1245us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 05:24:16.536106 (+ 1202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 05:24:16.536112 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 05:24:16.536113 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 05:24:16.536313 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 05:24:16.536551 (+ 238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 05:24:16.536552 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 05:24:16.536552 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 05:24:16.536638 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 05:24:16.536932 (+ 294us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 05:24:16.536938 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 05:24:16.537050 (+ 112us) client_negotiation.cc:769] Sending connection context
0812 05:24:16.537132 (+ 82us) client_negotiation.cc:240] Negotiation successful
0812 05:24:16.537247 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":76,"threads_started":1}
I20260812 05:24:16.537827 31297 heartbeater.cc:347] Connected to a master server at 127.31.100.62:40857
I20260812 05:24:16.537905 31297 heartbeater.cc:464] Registering TS with master...
I20260812 05:24:16.538066 31297 heartbeater.cc:511] Master 127.31.100.62:40857 requested a full tablet report, sending...
I20260812 05:24:16.538371 30537 ts_manager.cc:194] Registered new tserver with Master: 8d5863a178ae414f897ea77e4dc693ea (127.31.100.3:33651)
I20260812 05:24:16.538702 30537 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.100.3@KRBTEST.COM'} at 127.31.100.3:35607
I20260812 05:24:16.542928 32144 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 05:24:16.549191 30608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:24:16.544108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55572 (local address 127.31.100.62:40857)
0812 05:24:16.544157 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0812 05:24:16.544159 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 05:24:16.544246 (+ 87us) server_negotiation.cc:407] Connection header received
0812 05:24:16.544288 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:24:16.544288 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:24:16.544296 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:24:16.544340 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:24:16.544815 (+ 475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:24:16.545248 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:24:16.545939 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:24:16.546111 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:24:16.546155 (+ 44us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:24:16.547224 (+ 1069us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 05:24:16.547225 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 05:24:16.547225 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 05:24:16.547231 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0812 05:24:16.548370 (+ 1139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:24:16.548663 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:24:16.548663 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:24:16.548664 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:24:16.548707 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:24:16.548837 (+ 130us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:24:16.548837 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:24:16.548837 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:24:16.548940 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 05:24:16.549008 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0812 05:24:16.549089 (+ 81us) server_negotiation.cc:299] Negotiation successful
0812 05:24:16.549136 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260812 05:24:16.550496 30537 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55572:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 05:24:16.551957 30537 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 05:24:16.557925 31316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:24:16.554235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.2:41199 (local address 127.0.0.1:57174)
0812 05:24:16.554508 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0812 05:24:16.554517 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0812 05:24:16.554558 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:24:16.554898 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:24:16.554899 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:24:16.554901 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 05:24:16.554937 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:24:16.554938 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:24:16.555877 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:24:16.555878 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:24:16.556727 (+ 849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:24:16.556730 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:24:16.557649 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:24:16.557650 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:24:16.557736 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:24:16.557749 (+ 13us) client_negotiation.cc:769] Sending connection context
0812 05:24:16.557809 (+ 60us) client_negotiation.cc:240] Negotiation successful
0812 05:24:16.557857 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":34,"threads_started":1}
I20260812 05:24:16.558158 31312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:24:16.554000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39134 (local address 127.31.100.1:39749)
0812 05:24:16.554201 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0812 05:24:16.554205 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 05:24:16.554217 (+ 12us) server_negotiation.cc:407] Connection header received
0812 05:24:16.554260 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:24:16.554262 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:24:16.554278 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:24:16.554431 (+ 153us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 05:24:16.554575 (+ 144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:24:16.555675 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:24:16.556986 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:24:16.557723 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:24:16.557780 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:24:16.557828 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0812 05:24:16.557934 (+ 106us) server_negotiation.cc:299] Negotiation successful
0812 05:24:16.557970 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":170,"thread_start_us":95,"threads_started":1}
I20260812 05:24:16.558269 31317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:24:16.554338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57174 (local address 127.31.100.2:41199)
0812 05:24:16.554661 (+ 323us) server_negotiation.cc:206] Beginning negotiation
0812 05:24:16.554665 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 05:24:16.554675 (+ 10us) server_negotiation.cc:407] Connection header received
0812 05:24:16.554721 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:24:16.554723 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:24:16.554737 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:24:16.554807 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 05:24:16.555014 (+ 207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:24:16.555777 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:24:16.556811 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:24:16.557558 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:24:16.557627 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:24:16.557691 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0812 05:24:16.558079 (+ 388us) server_negotiation.cc:299] Negotiation successful
0812 05:24:16.558130 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":286,"thread_start_us":81,"threads_started":1}
I20260812 05:24:16.558030 31310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:24:16.553896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.1:39749 (local address 127.0.0.1:39134)
0812 05:24:16.554004 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0812 05:24:16.554040 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0812 05:24:16.554096 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:24:16.554433 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:24:16.554436 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:24:16.554439 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 05:24:16.554500 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:24:16.554503 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:24:16.555782 (+ 1279us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:24:16.555783 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:24:16.556907 (+ 1124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:24:16.556910 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:24:16.557783 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:24:16.557784 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:24:16.557847 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:24:16.557852 (+ 5us) client_negotiation.cc:769] Sending connection context
0812 05:24:16.557915 (+ 63us) client_negotiation.cc:240] Negotiation successful
0812 05:24:16.557957 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":86,"mutex_wait_us":35,"thread_start_us":57,"threads_started":1}
I20260812 05:24:16.558429 31315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:24:16.554173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43986 (local address 127.31.100.3:33651)
0812 05:24:16.554313 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0812 05:24:16.554317 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 05:24:16.554412 (+ 95us) server_negotiation.cc:407] Connection header received
0812 05:24:16.554515 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:24:16.554517 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:24:16.554531 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:24:16.554578 (+ 47us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 05:24:16.554776 (+ 198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:24:16.556117 (+ 1341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:24:16.557430 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:24:16.558114 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:24:16.558182 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:24:16.558234 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0812 05:24:16.558319 (+ 85us) server_negotiation.cc:299] Negotiation successful
0812 05:24:16.558362 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":66,"threads_started":1}
I20260812 05:24:16.558450 31314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:24:16.554107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.3:33651 (local address 127.0.0.1:43986)
0812 05:24:16.554358 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0812 05:24:16.554367 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0812 05:24:16.554438 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:24:16.554660 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:24:16.554661 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:24:16.554663 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 05:24:16.554708 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:24:16.554710 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:24:16.556214 (+ 1504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:24:16.556215 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:24:16.557359 (+ 1144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:24:16.557361 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:24:16.558193 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:24:16.558194 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:24:16.558253 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:24:16.558257 (+ 4us) client_negotiation.cc:769] Sending connection context
0812 05:24:16.558308 (+ 51us) client_negotiation.cc:240] Negotiation successful
0812 05:24:16.558342 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":50,"threads_started":1}
I20260812 05:24:16.559110 30986 tablet_service.cc:1467] Processing CreateTablet for tablet bff0bcb9e9f8478992257ea808d3a355 (DEFAULT_TABLE table=test-table [id=7a116de1c2e749528c17fa890598d7a1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 05:24:16.559123 30744 tablet_service.cc:1467] Processing CreateTablet for tablet bff0bcb9e9f8478992257ea808d3a355 (DEFAULT_TABLE table=test-table [id=7a116de1c2e749528c17fa890598d7a1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 05:24:16.559443 30744 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bff0bcb9e9f8478992257ea808d3a355. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 05:24:16.559464 30986 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bff0bcb9e9f8478992257ea808d3a355. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 05:24:16.561879 31201 tablet_service.cc:1467] Processing CreateTablet for tablet bff0bcb9e9f8478992257ea808d3a355 (DEFAULT_TABLE table=test-table [id=7a116de1c2e749528c17fa890598d7a1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 05:24:16.562178 31201 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bff0bcb9e9f8478992257ea808d3a355. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 05:24:16.562208 31321 tablet_bootstrap.cc:492] T bff0bcb9e9f8478992257ea808d3a355 P 00f8cc2a396a4a43becb5e9ec2ab21a8: Bootstrap starting.
I20260812 05:24:16.562719 31321 tablet_bootstrap.cc:654] T bff0bcb9e9f8478992257ea808d3a355 P 00f8cc2a396a4a43becb5e9ec2ab21a8: Neither blocks nor log segments found. Creating new log.
I20260812 05:24:16.563134 31321 log.cc:824] T bff0bcb9e9f8478992257ea808d3a355 P 00f8cc2a396a4a43becb5e9ec2ab21a8: Log is configured to *not* fsync() on all Append() calls
I20260812 05:24:16.563438 31322 tablet_bootstrap.cc:492] T bff0bcb9e9f8478992257ea808d3a355 P 0d6f22f2631e4e6aadad889de48ea392: Bootstrap starting.
I20260812 05:24:16.563968 31322 tablet_bootstrap.cc:654] T bff0bcb9e9f8478992257ea808d3a355 P 0d6f22f2631e4e6aadad889de48ea392: Neither blocks nor log segments found. Creating new log.
I20260812 05:24:16.564373 31322 log.cc:824] T bff0bcb9e9f8478992257ea808d3a355 P 0d6f22f2631e4e6aadad889de48ea392: Log is configured to *not* fsync() on all Append() calls
I20260812 05:24:16.564533 31327 tablet_bootstrap.cc:492] T bff0bcb9e9f8478992257ea808d3a355 P 8d5863a178ae414f897ea77e4dc693ea: Bootstrap starting.
I20260812 05:24:16.564849 31321 tablet_bootstrap.cc:492] T bff0bcb9e9f8478992257ea808d3a355 P 00f8cc2a396a4a43becb5e9ec2ab21a8: No bootstrap required, opened a new log
I20260812 05:24:16.564915 31321 ts_tablet_manager.cc:1397] T bff0bcb9e9f8478992257ea808d3a355 P 00f8cc2a396a4a43becb5e9ec2ab21a8: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260812 05:24:16.565225 31327 tablet_bootstrap.cc:654] T bff0bcb9e9f8478992257ea808d3a355 P 8d5863a178ae414f897ea77e4dc693ea: Neither blocks nor log segments found. Creating new log.
I20260812 05:24:16.565726 31327 log.cc:824] T bff0bcb9e9f8478992257ea808d3a355 P 8d5863a178ae414f897ea77e4dc693ea: Log is configured to *not* fsync() on all Append() calls
I20260812 05:24:16.566290 31327 tablet_bootstrap.cc:492] T bff0bcb9e9f8478992257ea808d3a355 P 8d5863a178ae414f897ea77e4dc693ea: No bootstrap required, opened a new log
I20260812 05:24:16.566349 31327 ts_tablet_manager.cc:1397] T bff0bcb9e9f8478992257ea808d3a355 P 8d5863a178ae414f897ea77e4dc693ea: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260812 05:24:16.566478 31321 raft_consensus.cc:348] T bff0bcb9e9f8478992257ea808d3a355 P 00f8cc2a396a4a43becb5e9ec2ab21a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d6f22f2631e4e6aadad889de48ea392" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 39749 } } peers { permanent_uuid: "8d5863a178ae414f897ea77e4dc693ea" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 33651 } } peers { permanent_uuid: "00f8cc2a396a4a43becb5e9ec2ab21a8" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 41199 } }
I20260812 05:24:16.566612 31321 raft_consensus.cc:374] T bff0bcb9e9f8478992257ea808d3a355 P 00f8cc2a396a4a43becb5e9ec2ab21a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 05:24:16.566632 31321 raft_consensus.cc:729] T bff0bcb9e9f8478992257ea808d3a355 P 00f8cc2a396a4a43becb5e9ec2ab21a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00f8cc2a396a4a43becb5e9ec2ab21a8, State: Initialized, Role: FOLLOWER
I20260812 05:24:16.566742 31321 consensus_queue.cc:260] T bff0bcb9e9f8478992257ea808d3a355 P 00f8cc2a396a4a43becb5e9ec2ab21a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d6f22f2631e4e6aadad889de48ea392" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 39749 } } peers { permanent_uuid: "8d5863a178ae414f897ea77e4dc693ea" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 33651 } } peers { permanent_uuid: "00f8cc2a396a4a43becb5e9ec2ab21a8" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 41199 } }
I20260812 05:24:16.566972 31321 ts_tablet_manager.cc:1428] T bff0bcb9e9f8478992257ea808d3a355 P 00f8cc2a396a4a43becb5e9ec2ab21a8: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260812 05:24:16.566998 31083 heartbeater.cc:503] Master 127.31.100.62:40857 was elected leader, sending a full tablet report...
I20260812 05:24:16.567286 31322 tablet_bootstrap.cc:492] T bff0bcb9e9f8478992257ea808d3a355 P 0d6f22f2631e4e6aadad889de48ea392: No bootstrap required, opened a new log
I20260812 05:24:16.567353 31322 ts_tablet_manager.cc:1397] T bff0bcb9e9f8478992257ea808d3a355 P 0d6f22f2631e4e6aadad889de48ea392: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260812 05:24:16.568289 31327 raft_consensus.cc:348] T bff0bcb9e9f8478992257ea808d3a355 P 8d5863a178ae414f897ea77e4dc693ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d6f22f2631e4e6aadad889de48ea392" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 39749 } } peers { permanent_uuid: "8d5863a178ae414f897ea77e4dc693ea" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 33651 } } peers { permanent_uuid: "00f8cc2a396a4a43becb5e9ec2ab21a8" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 41199 } }
I20260812 05:24:16.568418 31327 raft_consensus.cc:374] T bff0bcb9e9f8478992257ea808d3a355 P 8d5863a178ae414f897ea77e4dc693ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 05:24:16.568449 31327 raft_consensus.cc:729] T bff0bcb9e9f8478992257ea808d3a355 P 8d5863a178ae414f897ea77e4dc693ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d5863a178ae414f897ea77e4dc693ea, State: Initialized, Role: FOLLOWER
I20260812 05:24:16.568547 31327 consensus_queue.cc:260] T bff0bcb9e9f8478992257ea808d3a355 P 8d5863a178ae414f897ea77e4dc693ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d6f22f2631e4e6aadad889de48ea392" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 39749 } } peers { permanent_uuid: "8d5863a178ae414f897ea77e4dc693ea" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 33651 } } peers { permanent_uuid: "00f8cc2a396a4a43becb5e9ec2ab21a8" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 41199 } }
I20260812 05:24:16.568775 31327 ts_tablet_manager.cc:1428] T bff0bcb9e9f8478992257ea808d3a355 P 8d5863a178ae414f897ea77e4dc693ea: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260812 05:24:16.568795 31297 heartbeater.cc:503] Master 127.31.100.62:40857 was elected leader, sending a full tablet report...
I20260812 05:24:16.568994 31322 raft_consensus.cc:348] T bff0bcb9e9f8478992257ea808d3a355 P 0d6f22f2631e4e6aadad889de48ea392 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d6f22f2631e4e6aadad889de48ea392" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 39749 } } peers { permanent_uuid: "8d5863a178ae414f897ea77e4dc693ea" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 33651 } } peers { permanent_uuid: "00f8cc2a396a4a43becb5e9ec2ab21a8" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 41199 } }
I20260812 05:24:16.569154 31322 raft_consensus.cc:374] T bff0bcb9e9f8478992257ea808d3a355 P 0d6f22f2631e4e6aadad889de48ea392 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 05:24:16.569195 31322 raft_consensus.cc:729] T bff0bcb9e9f8478992257ea808d3a355 P 0d6f22f2631e4e6aadad889de48ea392 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d6f22f2631e4e6aadad889de48ea392, State: Initialized, Role: FOLLOWER
I20260812 05:24:16.569294 31322 consensus_queue.cc:260] T bff0bcb9e9f8478992257ea808d3a355 P 0d6f22f2631e4e6aadad889de48ea392 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d6f22f2631e4e6aadad889de48ea392" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 39749 } } peers { permanent_uuid: "8d5863a178ae414f897ea77e4dc693ea" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 33651 } } peers { permanent_uuid: "00f8cc2a396a4a43becb5e9ec2ab21a8" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 41199 } }
I20260812 05:24:16.569583 31322 ts_tablet_manager.cc:1428] T bff0bcb9e9f8478992257ea808d3a355 P 0d6f22f2631e4e6aadad889de48ea392: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260812 05:24:16.569615 30862 heartbeater.cc:503] Master 127.31.100.62:40857 was elected leader, sending a full tablet report...
W20260812 05:24:16.638552 31088 tablet.cc:2369] T bff0bcb9e9f8478992257ea808d3a355 P 00f8cc2a396a4a43becb5e9ec2ab21a8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 05:24:16.671373 31330 raft_consensus.cc:482] T bff0bcb9e9f8478992257ea808d3a355 P 00f8cc2a396a4a43becb5e9ec2ab21a8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 05:24:16.671550 31330 raft_consensus.cc:504] T bff0bcb9e9f8478992257ea808d3a355 P 00f8cc2a396a4a43becb5e9ec2ab21a8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d6f22f2631e4e6aadad889de48ea392" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 39749 } } peers { permanent_uuid: "8d5863a178ae414f897ea77e4dc693ea" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 33651 } } peers { permanent_uuid: "00f8cc2a396a4a43becb5e9ec2ab21a8" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 41199 } }
I20260812 05:24:16.671896 31330 leader_election.cc:288] T bff0bcb9e9f8478992257ea808d3a355 P 00f8cc2a396a4a43becb5e9ec2ab21a8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0d6f22f2631e4e6aadad889de48ea392 (127.31.100.1:39749), 8d5863a178ae414f897ea77e4dc693ea (127.31.100.3:33651)
I20260812 05:24:16.674793 31315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:24:16.672232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.100.2:43277 (local address 127.31.100.3:33651)
0812 05:24:16.672282 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0812 05:24:16.672285 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 05:24:16.672391 (+ 106us) server_negotiation.cc:407] Connection header received
0812 05:24:16.672437 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:24:16.672438 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:24:16.672447 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:24:16.672523 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 05:24:16.672668 (+ 145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:24:16.673123 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:24:16.674103 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:24:16.674478 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:24:16.674537 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:24:16.674546 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0812 05:24:16.674683 (+ 137us) server_negotiation.cc:299] Negotiation successful
0812 05:24:16.674724 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260812 05:24:16.674815 31090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:24:16.672047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.1:39749 (local address 127.31.100.2:48901)
0812 05:24:16.672106 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0812 05:24:16.672121 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 05:24:16.672175 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:24:16.672367 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:24:16.672368 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:24:16.672370 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 05:24:16.672442 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:24:16.672444 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:24:16.673139 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:24:16.673140 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:24:16.674017 (+ 877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:24:16.674019 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:24:16.674563 (+ 544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:24:16.674564 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:24:16.674645 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:24:16.674648 (+ 3us) client_negotiation.cc:769] Sending connection context
0812 05:24:16.674700 (+ 52us) client_negotiation.cc:240] Negotiation successful
0812 05:24:16.674736 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36,"mutex_wait_us":40}
I20260812 05:24:16.674814 31312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:24:16.672153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.100.2:48901 (local address 127.31.100.1:39749)
0812 05:24:16.672219 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0812 05:24:16.672221 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 05:24:16.672228 (+ 7us) server_negotiation.cc:407] Connection header received
0812 05:24:16.672256 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:24:16.672257 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:24:16.672267 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:24:16.672318 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 05:24:16.672523 (+ 205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:24:16.673063 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:24:16.674099 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:24:16.674489 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:24:16.674557 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:24:16.674565 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0812 05:24:16.674712 (+ 147us) server_negotiation.cc:299] Negotiation successful
0812 05:24:16.674753 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260812 05:24:16.674820 31334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:24:16.672156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.3:33651 (local address 127.31.100.2:43277)
0812 05:24:16.672322 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0812 05:24:16.672331 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0812 05:24:16.672388 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:24:16.672536 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:24:16.672537 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:24:16.672538 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 05:24:16.672606 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:24:16.672608 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:24:16.673201 (+ 593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:24:16.673202 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:24:16.674021 (+ 819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:24:16.674023 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:24:16.674559 (+ 536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:24:16.674560 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:24:16.674622 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:24:16.674626 (+ 4us) client_negotiation.cc:769] Sending connection context
0812 05:24:16.674666 (+ 40us) client_negotiation.cc:240] Negotiation successful
0812 05:24:16.674703 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":112,"threads_started":1}
I20260812 05:24:16.675197 30780 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bff0bcb9e9f8478992257ea808d3a355" candidate_uuid: "00f8cc2a396a4a43becb5e9ec2ab21a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d6f22f2631e4e6aadad889de48ea392" is_pre_election: true
I20260812 05:24:16.675330 30780 raft_consensus.cc:2454] T bff0bcb9e9f8478992257ea808d3a355 P 0d6f22f2631e4e6aadad889de48ea392 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00f8cc2a396a4a43becb5e9ec2ab21a8 in term 0.
I20260812 05:24:16.675398 31226 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bff0bcb9e9f8478992257ea808d3a355" candidate_uuid: "00f8cc2a396a4a43becb5e9ec2ab21a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d5863a178ae414f897ea77e4dc693ea" is_pre_election: true
I20260812 05:24:16.675549 31226 raft_consensus.cc:2454] T bff0bcb9e9f8478992257ea808d3a355 P 8d5863a178ae414f897ea77e4dc693ea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00f8cc2a396a4a43becb5e9ec2ab21a8 in term 0.
I20260812 05:24:16.675572 30900 leader_election.cc:302] T bff0bcb9e9f8478992257ea808d3a355 P 00f8cc2a396a4a43becb5e9ec2ab21a8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00f8cc2a396a4a43becb5e9ec2ab21a8, 0d6f22f2631e4e6aadad889de48ea392; no voters:
I20260812 05:24:16.675694 31330 raft_consensus.cc:2781] T bff0bcb9e9f8478992257ea808d3a355 P 00f8cc2a396a4a43becb5e9ec2ab21a8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 05:24:16.675750 31330 raft_consensus.cc:482] T bff0bcb9e9f8478992257ea808d3a355 P 00f8cc2a396a4a43becb5e9ec2ab21a8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 05:24:16.675774 31330 raft_consensus.cc:3037] T bff0bcb9e9f8478992257ea808d3a355 P 00f8cc2a396a4a43becb5e9ec2ab21a8 [term 0 FOLLOWER]: Advancing to term 1
I20260812 05:24:16.676470 31330 raft_consensus.cc:504] T bff0bcb9e9f8478992257ea808d3a355 P 00f8cc2a396a4a43becb5e9ec2ab21a8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d6f22f2631e4e6aadad889de48ea392" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 39749 } } peers { permanent_uuid: "8d5863a178ae414f897ea77e4dc693ea" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 33651 } } peers { permanent_uuid: "00f8cc2a396a4a43becb5e9ec2ab21a8" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 41199 } }
I20260812 05:24:16.676582 31330 leader_election.cc:288] T bff0bcb9e9f8478992257ea808d3a355 P 00f8cc2a396a4a43becb5e9ec2ab21a8 [CANDIDATE]: Term 1 election: Requested vote from peers 0d6f22f2631e4e6aadad889de48ea392 (127.31.100.1:39749), 8d5863a178ae414f897ea77e4dc693ea (127.31.100.3:33651)
I20260812 05:24:16.676716 30780 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bff0bcb9e9f8478992257ea808d3a355" candidate_uuid: "00f8cc2a396a4a43becb5e9ec2ab21a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d6f22f2631e4e6aadad889de48ea392"
I20260812 05:24:16.676757 31226 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bff0bcb9e9f8478992257ea808d3a355" candidate_uuid: "00f8cc2a396a4a43becb5e9ec2ab21a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d5863a178ae414f897ea77e4dc693ea"
I20260812 05:24:16.676792 30780 raft_consensus.cc:3037] T bff0bcb9e9f8478992257ea808d3a355 P 0d6f22f2631e4e6aadad889de48ea392 [term 0 FOLLOWER]: Advancing to term 1
I20260812 05:24:16.676831 31226 raft_consensus.cc:3037] T bff0bcb9e9f8478992257ea808d3a355 P 8d5863a178ae414f897ea77e4dc693ea [term 0 FOLLOWER]: Advancing to term 1
I20260812 05:24:16.677531 30780 raft_consensus.cc:2454] T bff0bcb9e9f8478992257ea808d3a355 P 0d6f22f2631e4e6aadad889de48ea392 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00f8cc2a396a4a43becb5e9ec2ab21a8 in term 1.
I20260812 05:24:16.677575 31226 raft_consensus.cc:2454] T bff0bcb9e9f8478992257ea808d3a355 P 8d5863a178ae414f897ea77e4dc693ea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00f8cc2a396a4a43becb5e9ec2ab21a8 in term 1.
I20260812 05:24:16.677680 30900 leader_election.cc:302] T bff0bcb9e9f8478992257ea808d3a355 P 00f8cc2a396a4a43becb5e9ec2ab21a8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00f8cc2a396a4a43becb5e9ec2ab21a8, 0d6f22f2631e4e6aadad889de48ea392; no voters:
I20260812 05:24:16.677775 31330 raft_consensus.cc:2781] T bff0bcb9e9f8478992257ea808d3a355 P 00f8cc2a396a4a43becb5e9ec2ab21a8 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 05:24:16.677927 31330 raft_consensus.cc:686] T bff0bcb9e9f8478992257ea808d3a355 P 00f8cc2a396a4a43becb5e9ec2ab21a8 [term 1 LEADER]: Becoming Leader. State: Replica: 00f8cc2a396a4a43becb5e9ec2ab21a8, State: Running, Role: LEADER
I20260812 05:24:16.678023 31330 consensus_queue.cc:237] T bff0bcb9e9f8478992257ea808d3a355 P 00f8cc2a396a4a43becb5e9ec2ab21a8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d6f22f2631e4e6aadad889de48ea392" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 39749 } } peers { permanent_uuid: "8d5863a178ae414f897ea77e4dc693ea" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 33651 } } peers { permanent_uuid: "00f8cc2a396a4a43becb5e9ec2ab21a8" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 41199 } }
I20260812 05:24:16.678788 30537 catalog_manager.cc:5495] T bff0bcb9e9f8478992257ea808d3a355 P 00f8cc2a396a4a43becb5e9ec2ab21a8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 00f8cc2a396a4a43becb5e9ec2ab21a8 (127.31.100.2). New cstate: current_term: 1 leader_uuid: "00f8cc2a396a4a43becb5e9ec2ab21a8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d6f22f2631e4e6aadad889de48ea392" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 39749 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8d5863a178ae414f897ea77e4dc693ea" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 33651 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "00f8cc2a396a4a43becb5e9ec2ab21a8" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 41199 } health_report { overall_health: HEALTHY } } }
I20260812 05:24:16.700831 31317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:24:16.698821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57190 (local address 127.31.100.2:41199)
0812 05:24:16.698865 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0812 05:24:16.698866 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0812 05:24:16.698897 (+ 31us) server_negotiation.cc:407] Connection header received
0812 05:24:16.698927 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:24:16.698927 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:24:16.698942 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:24:16.698992 (+ 50us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 05:24:16.699098 (+ 106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:24:16.699548 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:24:16.700225 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:24:16.700396 (+ 171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:24:16.700436 (+ 40us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:24:16.700562 (+ 126us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 05:24:16.700640 (+ 78us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 05:24:16.700703 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0812 05:24:16.700770 (+ 67us) server_negotiation.cc:299] Negotiation successful
0812 05:24:16.700779 (+ 9us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260812 05:24:16.702515 31226 raft_consensus.cc:1261] T bff0bcb9e9f8478992257ea808d3a355 P 8d5863a178ae414f897ea77e4dc693ea [term 1 FOLLOWER]: Refusing update from remote peer 00f8cc2a396a4a43becb5e9ec2ab21a8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 05:24:16.702522 30780 raft_consensus.cc:1261] T bff0bcb9e9f8478992257ea808d3a355 P 0d6f22f2631e4e6aadad889de48ea392 [term 1 FOLLOWER]: Refusing update from remote peer 00f8cc2a396a4a43becb5e9ec2ab21a8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 05:24:16.702834 31330 consensus_queue.cc:1035] T bff0bcb9e9f8478992257ea808d3a355 P 00f8cc2a396a4a43becb5e9ec2ab21a8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d6f22f2631e4e6aadad889de48ea392" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 39749 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 05:24:16.702970 31335 consensus_queue.cc:1035] T bff0bcb9e9f8478992257ea808d3a355 P 00f8cc2a396a4a43becb5e9ec2ab21a8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d5863a178ae414f897ea77e4dc693ea" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 33651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 05:24:16.705159 31340 mvcc.cc:205] Tried to move back new op lower bound from 7317554203451047936 to 7317554203353694208. Current Snapshot: MvccSnapshot[applied={T|T < 7317554203451047936}]
I20260812 05:24:16.705312 31342 mvcc.cc:205] Tried to move back new op lower bound from 7317554203451047936 to 7317554203353694208. Current Snapshot: MvccSnapshot[applied={T|T < 7317554203451047936}]
I20260812 05:24:16.707393 30535 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55572:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 05:24:16.707499 30535 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55572:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 05:24:16.708184 30535 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7a40e95cce1d460485805b5d07145b95: Sending DeleteTablet for 3 replicas of tablet bff0bcb9e9f8478992257ea808d3a355
I20260812 05:24:16.708492 30744 tablet_service.cc:1514] Processing DeleteTablet for tablet bff0bcb9e9f8478992257ea808d3a355 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 05:24:16 UTC) from {username='kudu', principal='kudu/127.31.100.62@KRBTEST.COM'} at 127.0.0.1:39134
I20260812 05:24:16.708556 30986 tablet_service.cc:1514] Processing DeleteTablet for tablet bff0bcb9e9f8478992257ea808d3a355 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 05:24:16 UTC) from {username='kudu', principal='kudu/127.31.100.62@KRBTEST.COM'} at 127.0.0.1:57174
I20260812 05:24:16.708736 31201 tablet_service.cc:1514] Processing DeleteTablet for tablet bff0bcb9e9f8478992257ea808d3a355 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 05:24:16 UTC) from {username='kudu', principal='kudu/127.31.100.62@KRBTEST.COM'} at 127.0.0.1:43986
I20260812 05:24:16.708777 31356 tablet_replica.cc:330] T bff0bcb9e9f8478992257ea808d3a355 P 00f8cc2a396a4a43becb5e9ec2ab21a8: stopping tablet replica
I20260812 05:24:16.708865 31356 raft_consensus.cc:2229] T bff0bcb9e9f8478992257ea808d3a355 P 00f8cc2a396a4a43becb5e9ec2ab21a8 [term 1 LEADER]: Raft consensus shutting down.
I20260812 05:24:16.708958 31356 raft_consensus.cc:2258] T bff0bcb9e9f8478992257ea808d3a355 P 00f8cc2a396a4a43becb5e9ec2ab21a8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 05:24:16.709022 31355 tablet_replica.cc:330] T bff0bcb9e9f8478992257ea808d3a355 P 0d6f22f2631e4e6aadad889de48ea392: stopping tablet replica
I20260812 05:24:16.709100 32144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0waFvT/build/release/bin/kudu with pid 30640
I20260812 05:24:16.709115 31355 raft_consensus.cc:2229] T bff0bcb9e9f8478992257ea808d3a355 P 0d6f22f2631e4e6aadad889de48ea392 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 05:24:16.709203 31355 raft_consensus.cc:2258] T bff0bcb9e9f8478992257ea808d3a355 P 0d6f22f2631e4e6aadad889de48ea392 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 05:24:16.720742 31356 ts_tablet_manager.cc:1905] T bff0bcb9e9f8478992257ea808d3a355 P 00f8cc2a396a4a43becb5e9ec2ab21a8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 05:24:16.722371 31357 tablet_replica.cc:330] T bff0bcb9e9f8478992257ea808d3a355 P 8d5863a178ae414f897ea77e4dc693ea: stopping tablet replica
I20260812 05:24:16.722491 31357 raft_consensus.cc:2229] T bff0bcb9e9f8478992257ea808d3a355 P 8d5863a178ae414f897ea77e4dc693ea [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 05:24:16.722590 31357 raft_consensus.cc:2258] T bff0bcb9e9f8478992257ea808d3a355 P 8d5863a178ae414f897ea77e4dc693ea [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 05:24:16.722745 31356 ts_tablet_manager.cc:1918] T bff0bcb9e9f8478992257ea808d3a355 P 00f8cc2a396a4a43becb5e9ec2ab21a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 05:24:16.722955 31356 log.cc:1196] T bff0bcb9e9f8478992257ea808d3a355 P 00f8cc2a396a4a43becb5e9ec2ab21a8: Deleting WAL directory at /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786512017613366-32144-0/minicluster-data/ts-1/wal/wals/bff0bcb9e9f8478992257ea808d3a355
I20260812 05:24:16.723155 31357 ts_tablet_manager.cc:1905] T bff0bcb9e9f8478992257ea808d3a355 P 8d5863a178ae414f897ea77e4dc693ea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 05:24:16.723225 31356 ts_tablet_manager.cc:1939] T bff0bcb9e9f8478992257ea808d3a355 P 00f8cc2a396a4a43becb5e9ec2ab21a8: Deleting consensus metadata
I20260812 05:24:16.723609 30513 catalog_manager.cc:4841] TS 00f8cc2a396a4a43becb5e9ec2ab21a8 (127.31.100.2:41199): tablet bff0bcb9e9f8478992257ea808d3a355 (table test-table [id=7a116de1c2e749528c17fa890598d7a1]) successfully deleted
I20260812 05:24:16.724720 31357 ts_tablet_manager.cc:1918] T bff0bcb9e9f8478992257ea808d3a355 P 8d5863a178ae414f897ea77e4dc693ea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 05:24:16.724797 31357 log.cc:1196] T bff0bcb9e9f8478992257ea808d3a355 P 8d5863a178ae414f897ea77e4dc693ea: Deleting WAL directory at /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786512017613366-32144-0/minicluster-data/ts-2/wal/wals/bff0bcb9e9f8478992257ea808d3a355
I20260812 05:24:16.725078 31357 ts_tablet_manager.cc:1939] T bff0bcb9e9f8478992257ea808d3a355 P 8d5863a178ae414f897ea77e4dc693ea: Deleting consensus metadata
I20260812 05:24:16.725452 30513 catalog_manager.cc:4841] TS 8d5863a178ae414f897ea77e4dc693ea (127.31.100.3:33651): tablet bff0bcb9e9f8478992257ea808d3a355 (table test-table [id=7a116de1c2e749528c17fa890598d7a1]) successfully deleted
W20260812 05:24:16.725580 30513 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.31.100.1:39749 (error 108)
I20260812 05:24:16.725829 32144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0waFvT/build/release/bin/kudu with pid 30869
I20260812 05:24:16.725998 31314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:24:16.725875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.1:39749 (local address 127.0.0.1:39148)
0812 05:24:16.725929 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0812 05:24:16.725969 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.100.1:39749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
W20260812 05:24:16.743412 30513 catalog_manager.cc:4584] TS 0d6f22f2631e4e6aadad889de48ea392 (127.31.100.1:39749): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet bff0bcb9e9f8478992257ea808d3a355: Network error: Client connection negotiation failed: client connection to 127.31.100.1:39749: connect: Connection refused (error 111)
I20260812 05:24:16.747673 32144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0waFvT/build/release/bin/kudu with pid 31093
I20260812 05:24:16.756184 32144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0waFvT/build/release/bin/kudu with pid 30485
2026-08-12T05:24:16Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2129 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-08-12T05:24:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T05:24:16Z Disabled control of system clock
I20260812 05:24:16.791637 32144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0waFvT/build/release/bin/kudu
/tmp/dist-test-task0waFvT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786512017613366-32144-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786512017613366-32144-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786512017613366-32144-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786512017613366-32144-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.100.62:43985
--webserver_interface=127.31.100.62
--webserver_port=0
--builtin_ntp_servers=127.31.100.20:40125
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.100.62:43985
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260812 05:24:16.870023 31365 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 05:24:16.870224 31365 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 05:24:16.870244 31365 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 05:24:16.871294 31365 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 05:24:16.871325 31365 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 05:24:16.871337 31365 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 05:24:16.871348 31365 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 05:24:16.871359 31365 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 05:24:16.872609 31365 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.100.20:40125
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786512017613366-32144-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786512017613366-32144-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.100.62:43985
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.100.62:43985
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786512017613366-32144-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.100.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31365
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786512017613366-32144-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 04:20:34 UTC on ccycloud.quasar-phqdqp.root.comops.site
build id 4581871
I20260812 05:24:16.872788 31365 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 05:24:16.873009 31365 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 05:24:16.877655 31375 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 05:24:16.877648 31370 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 05:24:16.877655 31372 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 05:24:16.877776 31365 server_base.cc:1034] running on GCE node
I20260812 05:24:16.878086 31365 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 05:24:16.878334 31365 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 05:24:16.879477 31365 hybrid_clock.cc:648] HybridClock initialized: now 1786512256879462 us; error 29 us; skew 500 ppm
I20260812 05:24:16.880641 31365 webserver.cc:460] Webserver started at http://127.31.100.62:43799/ using document root <none> and password file <none>
I20260812 05:24:16.880867 31365 fs_manager.cc:359] Metadata directory not provided
I20260812 05:24:16.880914 31365 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 05:24:16.881039 31365 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 05:24:16.882050 31365 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786512017613366-32144-0/minicluster-data/master-0/data/instance:
uuid: "67ce6d1cc1784329a55ccc230853942d"
format_stamp: "Formatted at 2026-08-12 05:24:16 on dist-test-slave-42z9"
server_key: "b6e5a77884f3798e2a8f9ddd7ae5a9f7"
server_key_iv: "97660fbcce72dbfc664a6fa1461a552d"
server_key_version: "encryptionkey@0"
I20260812 05:24:16.882360 31365 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786512017613366-32144-0/minicluster-data/master-0/wal/instance:
uuid: "67ce6d1cc1784329a55ccc230853942d"
format_stamp: "Formatted at 2026-08-12 05:24:16 on dist-test-slave-42z9"
server_key: "b6e5a77884f3798e2a8f9ddd7ae5a9f7"
server_key_iv: "97660fbcce72dbfc664a6fa1461a552d"
server_key_version: "encryptionkey@0"
I20260812 05:24:16.883513 31365 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260812 05:24:16.884155 31381 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 05:24:16.884280 31365 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260812 05:24:16.884346 31365 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786512017613366-32144-0/minicluster-data/master-0/data,/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786512017613366-32144-0/minicluster-data/master-0/wal
uuid: "67ce6d1cc1784329a55ccc230853942d"
format_stamp: "Formatted at 2026-08-12 05:24:16 on dist-test-slave-42z9"
server_key: "b6e5a77884f3798e2a8f9ddd7ae5a9f7"
server_key_iv: "97660fbcce72dbfc664a6fa1461a552d"
server_key_version: "encryptionkey@0"
I20260812 05:24:16.884418 31365 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786512017613366-32144-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786512017613366-32144-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786512017613366-32144-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 05:24:16.898190 31365 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 05:24:16.898435 31365 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 05:24:16.898561 31365 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 05:24:16.902187 31365 rpc_server.cc:307] RPC server started. Bound to: 127.31.100.62:43985
I20260812 05:24:16.902218 31478 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.100.62:43985 every 8 connection(s)
I20260812 05:24:16.902503 31365 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786512017613366-32144-0/minicluster-data/master-0/data/info.pb
I20260812 05:24:16.903431 31479 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 05:24:16.905616 32144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0waFvT/build/release/bin/kudu as pid 31365
I20260812 05:24:16.905642 31479 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 67ce6d1cc1784329a55ccc230853942d: Bootstrap starting.
I20260812 05:24:16.905725 32144 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786512017613366-32144-0/minicluster-data/master-0/wal/instance
I20260812 05:24:16.905943 32144 external_mini_cluster.cc:1428] Setting key 9ccf8d52aed953a400a5b7f750cf83dd
I20260812 05:24:16.906325 31479 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 67ce6d1cc1784329a55ccc230853942d: Neither blocks nor log segments found. Creating new log.
I20260812 05:24:16.906677 31479 log.cc:824] T 00000000000000000000000000000000 P 67ce6d1cc1784329a55ccc230853942d: Log is configured to *not* fsync() on all Append() calls
I20260812 05:24:16.907392 31479 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 67ce6d1cc1784329a55ccc230853942d: No bootstrap required, opened a new log
I20260812 05:24:16.909344 31479 raft_consensus.cc:348] T 00000000000000000000000000000000 P 67ce6d1cc1784329a55ccc230853942d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67ce6d1cc1784329a55ccc230853942d" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 43985 } }
I20260812 05:24:16.909555 31479 raft_consensus.cc:374] T 00000000000000000000000000000000 P 67ce6d1cc1784329a55ccc230853942d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 05:24:16.909603 31479 raft_consensus.cc:729] T 00000000000000000000000000000000 P 67ce6d1cc1784329a55ccc230853942d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67ce6d1cc1784329a55ccc230853942d, State: Initialized, Role: FOLLOWER
I20260812 05:24:16.909722 31479 consensus_queue.cc:260] T 00000000000000000000000000000000 P 67ce6d1cc1784329a55ccc230853942d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67ce6d1cc1784329a55ccc230853942d" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 43985 } }
I20260812 05:24:16.909795 31479 raft_consensus.cc:388] T 00000000000000000000000000000000 P 67ce6d1cc1784329a55ccc230853942d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 05:24:16.909825 31479 raft_consensus.cc:482] T 00000000000000000000000000000000 P 67ce6d1cc1784329a55ccc230853942d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 05:24:16.909871 31479 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 67ce6d1cc1784329a55ccc230853942d [term 0 FOLLOWER]: Advancing to term 1
I20260812 05:24:16.910532 31479 raft_consensus.cc:504] T 00000000000000000000000000000000 P 67ce6d1cc1784329a55ccc230853942d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67ce6d1cc1784329a55ccc230853942d" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 43985 } }
I20260812 05:24:16.910653 31479 leader_election.cc:302] T 00000000000000000000000000000000 P 67ce6d1cc1784329a55ccc230853942d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 67ce6d1cc1784329a55ccc230853942d; no voters:
I20260812 05:24:16.910822 31479 leader_election.cc:288] T 00000000000000000000000000000000 P 67ce6d1cc1784329a55ccc230853942d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 05:24:16.910856 31488 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 67ce6d1cc1784329a55ccc230853942d [term 1 FOLLOWER]: Leader election won for term 1
I20260812 05:24:16.910962 31488 raft_consensus.cc:686] T 00000000000000000000000000000000 P 67ce6d1cc1784329a55ccc230853942d [term 1 LEADER]: Becoming Leader. State: Replica: 67ce6d1cc1784329a55ccc230853942d, State: Running, Role: LEADER
I20260812 05:24:16.911098 31479 sys_catalog.cc:564] T 00000000000000000000000000000000 P 67ce6d1cc1784329a55ccc230853942d [sys.catalog]: configured and running, proceeding with master startup.
I20260812 05:24:16.911080 31488 consensus_queue.cc:237] T 00000000000000000000000000000000 P 67ce6d1cc1784329a55ccc230853942d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67ce6d1cc1784329a55ccc230853942d" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 43985 } }
I20260812 05:24:16.911391 31484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:24:16.906461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39550 (local address 127.31.100.62:43985)
0812 05:24:16.906647 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0812 05:24:16.906654 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 05:24:16.906677 (+ 23us) server_negotiation.cc:407] Connection header received
0812 05:24:16.907072 (+ 395us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:24:16.907079 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:24:16.907133 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:24:16.907243 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:24:16.907504 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:24:16.908375 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:24:16.909476 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:24:16.909960 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:24:16.910054 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:24:16.910677 (+ 623us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 05:24:16.910682 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 05:24:16.910682 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 05:24:16.910878 (+ 196us) server_negotiation.cc:922] calling sasl_server_start()
0812 05:24:16.910919 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 05:24:16.910933 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 05:24:16.910995 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0812 05:24:16.911166 (+ 171us) server_negotiation.cc:299] Negotiation successful
0812 05:24:16.911248 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":71,"threads_started":1}
I20260812 05:24:16.911499 31488 sys_catalog.cc:455] T 00000000000000000000000000000000 P 67ce6d1cc1784329a55ccc230853942d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "67ce6d1cc1784329a55ccc230853942d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67ce6d1cc1784329a55ccc230853942d" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 43985 } } }
I20260812 05:24:16.911604 31488 sys_catalog.cc:458] T 00000000000000000000000000000000 P 67ce6d1cc1784329a55ccc230853942d [sys.catalog]: This master's current role is: LEADER
I20260812 05:24:16.911782 31488 sys_catalog.cc:455] T 00000000000000000000000000000000 P 67ce6d1cc1784329a55ccc230853942d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 67ce6d1cc1784329a55ccc230853942d. Latest consensus state: current_term: 1 leader_uuid: "67ce6d1cc1784329a55ccc230853942d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67ce6d1cc1784329a55ccc230853942d" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 43985 } } }
I20260812 05:24:16.911844 31488 sys_catalog.cc:458] T 00000000000000000000000000000000 P 67ce6d1cc1784329a55ccc230853942d [sys.catalog]: This master's current role is: LEADER
I20260812 05:24:16.911983 31502 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 05:24:16.912451 31502 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 05:24:16.913991 31502 catalog_manager.cc:1347] Generated new cluster ID: 00f8877617244c1c9b78473ad4b3b866
I20260812 05:24:16.914053 31502 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 05:24:16.927587 31502 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 05:24:16.928038 31502 catalog_manager.cc:1504] Loading token signing keys...
I20260812 05:24:16.933794 31502 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 67ce6d1cc1784329a55ccc230853942d: Generated new TSK 0
I20260812 05:24:16.933943 31502 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 05:24:16.939414 32144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0waFvT/build/release/bin/kudu
/tmp/dist-test-task0waFvT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786512017613366-32144-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786512017613366-32144-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786512017613366-32144-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786512017613366-32144-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.100.1:0
--local_ip_for_outbound_sockets=127.31.100.1
--webserver_interface=127.31.100.1
--webserver_port=0
--tserver_master_addrs=127.31.100.62:43985
--builtin_ntp_servers=127.31.100.20:40125
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260812 05:24:17.020779 31518 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 05:24:17.020972 31518 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 05:24:17.020991 31518 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 05:24:17.022119 31518 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 05:24:17.022156 31518 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 05:24:17.022179 31518 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.100.1
I20260812 05:24:17.023471 31518 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.100.20:40125
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786512017613366-32144-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786512017613366-32144-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.100.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786512017613366-32144-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.100.1
--webserver_port=0
--tserver_master_addrs=127.31.100.62:43985
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31518
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.100.1
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786512017613366-32144-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 04:20:34 UTC on ccycloud.quasar-phqdqp.root.comops.site
build id 4581871
I20260812 05:24:17.023693 31518 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 05:24:17.023974 31518 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 05:24:17.028716 31523 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 05:24:17.028738 31524 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 05:24:17.028860 31518 server_base.cc:1034] running on GCE node
W20260812 05:24:17.028731 31527 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 05:24:17.029138 31518 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 05:24:17.029381 31518 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 05:24:17.030548 31518 hybrid_clock.cc:648] HybridClock initialized: now 1786512257030533 us; error 31 us; skew 500 ppm
I20260812 05:24:17.031692 31518 webserver.cc:460] Webserver started at http://127.31.100.1:41465/ using document root <none> and password file <none>
I20260812 05:24:17.031903 31518 fs_manager.cc:359] Metadata directory not provided
I20260812 05:24:17.031961 31518 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 05:24:17.032087 31518 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 05:24:17.033102 31518 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786512017613366-32144-0/minicluster-data/ts-0/data/instance:
uuid: "73dd8c812ac3452a9f9a8822fc64620e"
format_stamp: "Formatted at 2026-08-12 05:24:17 on dist-test-slave-42z9"
server_key: "ed5a69fdc34146a74102e23ec86e445d"
server_key_iv: "deeef6f25c95fa9c74df354f8126e70a"
server_key_version: "encryptionkey@0"
I20260812 05:24:17.033423 31518 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786512017613366-32144-0/minicluster-data/ts-0/wal/instance:
uuid: "73dd8c812ac3452a9f9a8822fc64620e"
format_stamp: "Formatted at 2026-08-12 05:24:17 on dist-test-slave-42z9"
server_key: "ed5a69fdc34146a74102e23ec86e445d"
server_key_iv: "deeef6f25c95fa9c74df354f8126e70a"
server_key_version: "encryptionkey@0"
I20260812 05:24:17.034765 31518 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260812 05:24:17.035519 31540 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 05:24:17.035686 31518 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260812 05:24:17.035772 31518 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786512017613366-32144-0/minicluster-data/ts-0/data,/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786512017613366-32144-0/minicluster-data/ts-0/wal
uuid: "73dd8c812ac3452a9f9a8822fc64620e"
format_stamp: "Formatted at 2026-08-12 05:24:17 on dist-test-slave-42z9"
server_key: "ed5a69fdc34146a74102e23ec86e445d"
server_key_iv: "deeef6f25c95fa9c74df354f8126e70a"
server_key_version: "encryptionkey@0"
I20260812 05:24:17.035840 31518 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786512017613366-32144-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786512017613366-32144-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786512017613366-32144-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 05:24:17.066485 31518 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 05:24:17.066802 31518 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 05:24:17.066943 31518 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 05:24:17.067176 31518 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 05:24:17.067516 31518 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 05:24:17.067554 31518 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 05:24:17.067584 31518 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 05:24:17.067600 31518 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 05:24:17.073065 31518 rpc_server.cc:307] RPC server started. Bound to: 127.31.100.1:44891
I20260812 05:24:17.073110 31726 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.100.1:44891 every 8 connection(s)
I20260812 05:24:17.073427 31518 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786512017613366-32144-0/minicluster-data/ts-0/data/info.pb
I20260812 05:24:17.075199 32144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0waFvT/build/release/bin/kudu as pid 31518
I20260812 05:24:17.075412 32144 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786512017613366-32144-0/minicluster-data/ts-0/wal/instance
I20260812 05:24:17.075759 32144 external_mini_cluster.cc:1428] Setting key c77043d7e96b6c8d6b28c814e2446e77
I20260812 05:24:17.077165 32144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0waFvT/build/release/bin/kudu
/tmp/dist-test-task0waFvT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786512017613366-32144-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786512017613366-32144-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786512017613366-32144-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786512017613366-32144-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.100.2:0
--local_ip_for_outbound_sockets=127.31.100.2
--webserver_interface=127.31.100.2
--webserver_port=0
--tserver_master_addrs=127.31.100.62:43985
--builtin_ntp_servers=127.31.100.20:40125
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260812 05:24:17.083346 31484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:24:17.074172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.100.1:55179 (local address 127.31.100.62:43985)
0812 05:24:17.074237 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0812 05:24:17.074240 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 05:24:17.074456 (+ 216us) server_negotiation.cc:407] Connection header received
0812 05:24:17.075061 (+ 605us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:24:17.075062 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:24:17.075080 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:24:17.075153 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:24:17.075933 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:24:17.076706 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:24:17.078120 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:24:17.078408 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:24:17.078444 (+ 36us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:24:17.083083 (+ 4639us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 05:24:17.083085 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 05:24:17.083085 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 05:24:17.083090 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0812 05:24:17.083105 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 05:24:17.083109 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 05:24:17.083148 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0812 05:24:17.083237 (+ 89us) server_negotiation.cc:299] Negotiation successful
0812 05:24:17.083265 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260812 05:24:17.083564 31736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:24:17.074139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.62:43985 (local address 127.31.100.1:55179)
0812 05:24:17.074338 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0812 05:24:17.074372 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0812 05:24:17.074965 (+ 593us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:24:17.075226 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:24:17.075231 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:24:17.075241 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0812 05:24:17.075827 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:24:17.075834 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:24:17.076825 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:24:17.076826 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:24:17.078024 (+ 1198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:24:17.078028 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:24:17.078526 (+ 498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:24:17.078527 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:24:17.078674 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:24:17.082885 (+ 4211us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 05:24:17.082885 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0812 05:24:17.082935 (+ 50us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 05:24:17.082972 (+ 37us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 05:24:17.083162 (+ 190us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 05:24:17.083169 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 05:24:17.083172 (+ 3us) client_negotiation.cc:769] Sending connection context
0812 05:24:17.083237 (+ 65us) client_negotiation.cc:240] Negotiation successful
0812 05:24:17.083340 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":85,"threads_started":1}
I20260812 05:24:17.085333 31728 heartbeater.cc:347] Connected to a master server at 127.31.100.62:43985
I20260812 05:24:17.085467 31728 heartbeater.cc:464] Registering TS with master...
I20260812 05:24:17.085673 31728 heartbeater.cc:511] Master 127.31.100.62:43985 requested a full tablet report, sending...
I20260812 05:24:17.086344 31407 ts_manager.cc:194] Registered new tserver with Master: 73dd8c812ac3452a9f9a8822fc64620e (127.31.100.1:44891)
I20260812 05:24:17.087275 31407 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.31.100.1:55179
W20260812 05:24:17.164726 31738 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 05:24:17.164923 31738 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 05:24:17.164943 31738 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 05:24:17.166071 31738 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 05:24:17.166110 31738 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 05:24:17.166133 31738 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.100.2
I20260812 05:24:17.167426 31738 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.100.20:40125
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786512017613366-32144-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786512017613366-32144-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.100.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786512017613366-32144-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.100.2
--webserver_port=0
--tserver_master_addrs=127.31.100.62:43985
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31738
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.100.2
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786512017613366-32144-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 04:20:34 UTC on ccycloud.quasar-phqdqp.root.comops.site
build id 4581871
I20260812 05:24:17.167603 31738 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 05:24:17.167840 31738 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 05:24:17.172621 31749 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 05:24:17.172631 31754 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 05:24:17.172631 31752 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 05:24:17.172727 31738 server_base.cc:1034] running on GCE node
I20260812 05:24:17.173022 31738 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 05:24:17.173238 31738 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 05:24:17.174369 31738 hybrid_clock.cc:648] HybridClock initialized: now 1786512257174357 us; error 25 us; skew 500 ppm
I20260812 05:24:17.175458 31738 webserver.cc:460] Webserver started at http://127.31.100.2:36285/ using document root <none> and password file <none>
I20260812 05:24:17.175674 31738 fs_manager.cc:359] Metadata directory not provided
I20260812 05:24:17.175722 31738 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 05:24:17.175850 31738 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 05:24:17.176837 31738 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786512017613366-32144-0/minicluster-data/ts-1/data/instance:
uuid: "52d9d219ab104abca8578a3a296c8f51"
format_stamp: "Formatted at 2026-08-12 05:24:17 on dist-test-slave-42z9"
server_key: "e93227d8b0d30f659773e330eafc1e14"
server_key_iv: "83e5a7a08ef6d8c923e7fbc73d191afb"
server_key_version: "encryptionkey@0"
I20260812 05:24:17.177175 31738 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786512017613366-32144-0/minicluster-data/ts-1/wal/instance:
uuid: "52d9d219ab104abca8578a3a296c8f51"
format_stamp: "Formatted at 2026-08-12 05:24:17 on dist-test-slave-42z9"
server_key: "e93227d8b0d30f659773e330eafc1e14"
server_key_iv: "83e5a7a08ef6d8c923e7fbc73d191afb"
server_key_version: "encryptionkey@0"
I20260812 05:24:17.178421 31738 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260812 05:24:17.179075 31762 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 05:24:17.179219 31738 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260812 05:24:17.179283 31738 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786512017613366-32144-0/minicluster-data/ts-1/data,/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786512017613366-32144-0/minicluster-data/ts-1/wal
uuid: "52d9d219ab104abca8578a3a296c8f51"
format_stamp: "Formatted at 2026-08-12 05:24:17 on dist-test-slave-42z9"
server_key: "e93227d8b0d30f659773e330eafc1e14"
server_key_iv: "83e5a7a08ef6d8c923e7fbc73d191afb"
server_key_version: "encryptionkey@0"
I20260812 05:24:17.179333 31738 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786512017613366-32144-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786512017613366-32144-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786512017613366-32144-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 05:24:17.186986 31738 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 05:24:17.187251 31738 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 05:24:17.187371 31738 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 05:24:17.187575 31738 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 05:24:17.187911 31738 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 05:24:17.187947 31738 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 05:24:17.187973 31738 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 05:24:17.187999 31738 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 05:24:17.193434 31738 rpc_server.cc:307] RPC server started. Bound to: 127.31.100.2:37537
I20260812 05:24:17.193504 31952 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.100.2:37537 every 8 connection(s)
I20260812 05:24:17.193813 31738 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786512017613366-32144-0/minicluster-data/ts-1/data/info.pb
I20260812 05:24:17.195891 32144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0waFvT/build/release/bin/kudu as pid 31738
I20260812 05:24:17.195995 32144 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786512017613366-32144-0/minicluster-data/ts-1/wal/instance
I20260812 05:24:17.196190 32144 external_mini_cluster.cc:1428] Setting key c3180df29af9254fbd59c91ac0d6343e
I20260812 05:24:17.197168 32144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0waFvT/build/release/bin/kudu
/tmp/dist-test-task0waFvT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786512017613366-32144-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786512017613366-32144-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786512017613366-32144-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786512017613366-32144-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.100.3:0
--local_ip_for_outbound_sockets=127.31.100.3
--webserver_interface=127.31.100.3
--webserver_port=0
--tserver_master_addrs=127.31.100.62:43985
--builtin_ntp_servers=127.31.100.20:40125
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260812 05:24:17.199388 31484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:24:17.194458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.100.2:45229 (local address 127.31.100.62:43985)
0812 05:24:17.194514 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0812 05:24:17.194516 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 05:24:17.194728 (+ 212us) server_negotiation.cc:407] Connection header received
0812 05:24:17.195216 (+ 488us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:24:17.195218 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:24:17.195231 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:24:17.195285 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:24:17.196015 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:24:17.196713 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:24:17.198032 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:24:17.198225 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:24:17.198280 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:24:17.199125 (+ 845us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 05:24:17.199126 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 05:24:17.199127 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 05:24:17.199131 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0812 05:24:17.199151 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 05:24:17.199155 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 05:24:17.199204 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0812 05:24:17.199271 (+ 67us) server_negotiation.cc:299] Negotiation successful
0812 05:24:17.199299 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260812 05:24:17.199517 31955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:24:17.194448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.62:43985 (local address 127.31.100.2:45229)
0812 05:24:17.194619 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0812 05:24:17.194651 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0812 05:24:17.195123 (+ 472us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:24:17.195349 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:24:17.195352 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:24:17.195360 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0812 05:24:17.195904 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:24:17.195910 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:24:17.196806 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:24:17.196807 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:24:17.197949 (+ 1142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:24:17.197953 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:24:17.198299 (+ 346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:24:17.198301 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:24:17.198442 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:24:17.198985 (+ 543us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 05:24:17.198985 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0812 05:24:17.199019 (+ 34us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 05:24:17.199040 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 05:24:17.199214 (+ 174us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 05:24:17.199217 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 05:24:17.199219 (+ 2us) client_negotiation.cc:769] Sending connection context
0812 05:24:17.199258 (+ 39us) client_negotiation.cc:240] Negotiation successful
0812 05:24:17.199335 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":75,"threads_started":1}
I20260812 05:24:17.200022 31953 heartbeater.cc:347] Connected to a master server at 127.31.100.62:43985
I20260812 05:24:17.200102 31953 heartbeater.cc:464] Registering TS with master...
I20260812 05:24:17.200268 31953 heartbeater.cc:511] Master 127.31.100.62:43985 requested a full tablet report, sending...
I20260812 05:24:17.200601 31407 ts_manager.cc:194] Registered new tserver with Master: 52d9d219ab104abca8578a3a296c8f51 (127.31.100.2:37537)
I20260812 05:24:17.201049 31407 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.31.100.2:45229
W20260812 05:24:17.280033 31956 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 05:24:17.280221 31956 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 05:24:17.280241 31956 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 05:24:17.281318 31956 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 05:24:17.281347 31956 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 05:24:17.281370 31956 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.100.3
I20260812 05:24:17.282784 31956 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.100.20:40125
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786512017613366-32144-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786512017613366-32144-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.100.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786512017613366-32144-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.100.3
--webserver_port=0
--tserver_master_addrs=127.31.100.62:43985
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31956
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.100.3
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786512017613366-32144-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 04:20:34 UTC on ccycloud.quasar-phqdqp.root.comops.site
build id 4581871
I20260812 05:24:17.282969 31956 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 05:24:17.283197 31956 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 05:24:17.287894 31966 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 05:24:17.287971 31963 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 05:24:17.288149 31964 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 05:24:17.288285 31956 server_base.cc:1034] running on GCE node
I20260812 05:24:17.288517 31956 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 05:24:17.288726 31956 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 05:24:17.289852 31956 hybrid_clock.cc:648] HybridClock initialized: now 1786512257289841 us; error 30 us; skew 500 ppm
I20260812 05:24:17.291023 31956 webserver.cc:460] Webserver started at http://127.31.100.3:33353/ using document root <none> and password file <none>
I20260812 05:24:17.291240 31956 fs_manager.cc:359] Metadata directory not provided
I20260812 05:24:17.291286 31956 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 05:24:17.291401 31956 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 05:24:17.292402 31956 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786512017613366-32144-0/minicluster-data/ts-2/data/instance:
uuid: "725e2e2910cd4f1098e084c017ac2aeb"
format_stamp: "Formatted at 2026-08-12 05:24:17 on dist-test-slave-42z9"
server_key: "a62f2b074bb4757e8a892763244db8a4"
server_key_iv: "2c1288f2287fd23ad864ebd3a43d5b98"
server_key_version: "encryptionkey@0"
I20260812 05:24:17.292740 31956 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786512017613366-32144-0/minicluster-data/ts-2/wal/instance:
uuid: "725e2e2910cd4f1098e084c017ac2aeb"
format_stamp: "Formatted at 2026-08-12 05:24:17 on dist-test-slave-42z9"
server_key: "a62f2b074bb4757e8a892763244db8a4"
server_key_iv: "2c1288f2287fd23ad864ebd3a43d5b98"
server_key_version: "encryptionkey@0"
I20260812 05:24:17.294039 31956 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260812 05:24:17.294665 31976 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 05:24:17.294808 31956 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260812 05:24:17.294893 31956 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786512017613366-32144-0/minicluster-data/ts-2/data,/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786512017613366-32144-0/minicluster-data/ts-2/wal
uuid: "725e2e2910cd4f1098e084c017ac2aeb"
format_stamp: "Formatted at 2026-08-12 05:24:17 on dist-test-slave-42z9"
server_key: "a62f2b074bb4757e8a892763244db8a4"
server_key_iv: "2c1288f2287fd23ad864ebd3a43d5b98"
server_key_version: "encryptionkey@0"
I20260812 05:24:17.294950 31956 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786512017613366-32144-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786512017613366-32144-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786512017613366-32144-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 05:24:17.304387 31956 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 05:24:17.304627 31956 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 05:24:17.304740 31956 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 05:24:17.304919 31956 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 05:24:17.305202 31956 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 05:24:17.305243 31956 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 05:24:17.305269 31956 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 05:24:17.305294 31956 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 05:24:17.310506 31956 rpc_server.cc:307] RPC server started. Bound to: 127.31.100.3:46445
I20260812 05:24:17.310565 32176 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.100.3:46445 every 8 connection(s)
I20260812 05:24:17.310856 31956 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786512017613366-32144-0/minicluster-data/ts-2/data/info.pb
I20260812 05:24:17.313050 32144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0waFvT/build/release/bin/kudu as pid 31956
I20260812 05:24:17.313144 32144 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786512017613366-32144-0/minicluster-data/ts-2/wal/instance
I20260812 05:24:17.313313 32144 external_mini_cluster.cc:1428] Setting key 8c05012d619e5f54a0a30d490e67928e
I20260812 05:24:17.315716 31484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:24:17.311467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.100.3:49849 (local address 127.31.100.62:43985)
0812 05:24:17.311520 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0812 05:24:17.311522 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 05:24:17.311733 (+ 211us) server_negotiation.cc:407] Connection header received
0812 05:24:17.312208 (+ 475us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:24:17.312210 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:24:17.312222 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:24:17.312272 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:24:17.312886 (+ 614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:24:17.313501 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:24:17.314506 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:24:17.314709 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:24:17.314763 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:24:17.315458 (+ 695us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 05:24:17.315459 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 05:24:17.315460 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 05:24:17.315464 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0812 05:24:17.315480 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 05:24:17.315484 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 05:24:17.315529 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0812 05:24:17.315612 (+ 83us) server_negotiation.cc:299] Negotiation successful
0812 05:24:17.315639 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260812 05:24:17.315896 32179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:24:17.311455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.62:43985 (local address 127.31.100.3:49849)
0812 05:24:17.311630 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0812 05:24:17.311666 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0812 05:24:17.312117 (+ 451us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:24:17.312335 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:24:17.312338 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:24:17.312346 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0812 05:24:17.312803 (+ 457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:24:17.312807 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:24:17.313585 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:24:17.313586 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:24:17.314432 (+ 846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:24:17.314434 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:24:17.314775 (+ 341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:24:17.314776 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:24:17.314889 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:24:17.315335 (+ 446us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 05:24:17.315335 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0812 05:24:17.315361 (+ 26us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 05:24:17.315382 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 05:24:17.315538 (+ 156us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 05:24:17.315543 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 05:24:17.315546 (+ 3us) client_negotiation.cc:769] Sending connection context
0812 05:24:17.315608 (+ 62us) client_negotiation.cc:240] Negotiation successful
0812 05:24:17.315703 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":71,"threads_started":1}
I20260812 05:24:17.316267 32177 heartbeater.cc:347] Connected to a master server at 127.31.100.62:43985
I20260812 05:24:17.316351 32177 heartbeater.cc:464] Registering TS with master...
I20260812 05:24:17.316517 32177 heartbeater.cc:511] Master 127.31.100.62:43985 requested a full tablet report, sending...
I20260812 05:24:17.316817 31407 ts_manager.cc:194] Registered new tserver with Master: 725e2e2910cd4f1098e084c017ac2aeb (127.31.100.3:46445)
I20260812 05:24:17.317160 31407 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.31.100.3:49849
I20260812 05:24:17.324801 32144 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 05:24:17.328446 31484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:24:17.326113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39556 (local address 127.31.100.62:43985)
0812 05:24:17.326159 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0812 05:24:17.326160 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0812 05:24:17.326264 (+ 104us) server_negotiation.cc:407] Connection header received
0812 05:24:17.326304 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:24:17.326305 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:24:17.326313 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:24:17.326356 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:24:17.326470 (+ 114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:24:17.326912 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:24:17.327566 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:24:17.327729 (+ 163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:24:17.327769 (+ 40us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:24:17.328261 (+ 492us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 05:24:17.328262 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 05:24:17.328263 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 05:24:17.328266 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0812 05:24:17.328278 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 05:24:17.328281 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 05:24:17.328347 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0812 05:24:17.328387 (+ 40us) server_negotiation.cc:299] Negotiation successful
0812 05:24:17.328396 (+ 9us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260812 05:24:17.329782 31407 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39556:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 05:24:17.331259 31407 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 05:24:17.336910 32189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:24:17.333123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.1:44891 (local address 127.0.0.1:48440)
0812 05:24:17.333389 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0812 05:24:17.333397 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0812 05:24:17.333457 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:24:17.333767 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:24:17.333768 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:24:17.333770 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 05:24:17.333811 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:24:17.333813 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:24:17.335081 (+ 1268us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:24:17.335082 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:24:17.335821 (+ 739us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:24:17.335823 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:24:17.336619 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:24:17.336620 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:24:17.336699 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:24:17.336712 (+ 13us) client_negotiation.cc:769] Sending connection context
0812 05:24:17.336783 (+ 71us) client_negotiation.cc:240] Negotiation successful
0812 05:24:17.336836 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":102,"threads_started":1}
I20260812 05:24:17.336925 32197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:24:17.333361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48440 (local address 127.31.100.1:44891)
0812 05:24:17.333554 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0812 05:24:17.333558 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 05:24:17.333566 (+ 8us) server_negotiation.cc:407] Connection header received
0812 05:24:17.333615 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:24:17.333617 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:24:17.333633 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:24:17.333701 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 05:24:17.333888 (+ 187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:24:17.335000 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:24:17.335913 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:24:17.336532 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:24:17.336596 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:24:17.336688 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0812 05:24:17.336810 (+ 122us) server_negotiation.cc:299] Negotiation successful
0812 05:24:17.336847 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":91,"threads_started":1}
I20260812 05:24:17.337332 32188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:24:17.333123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.3:46445 (local address 127.0.0.1:34646)
0812 05:24:17.333632 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0812 05:24:17.333639 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0812 05:24:17.333681 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:24:17.333893 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:24:17.333893 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:24:17.333895 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 05:24:17.333946 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:24:17.333948 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:24:17.334870 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:24:17.334870 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:24:17.336129 (+ 1259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:24:17.336132 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:24:17.337120 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:24:17.337121 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:24:17.337181 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:24:17.337186 (+ 5us) client_negotiation.cc:769] Sending connection context
0812 05:24:17.337229 (+ 43us) client_negotiation.cc:240] Negotiation successful
0812 05:24:17.337267 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":500,"mutex_wait_us":2,"thread_start_us":45,"threads_started":1}
I20260812 05:24:17.337569 32192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:24:17.333255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34646 (local address 127.31.100.3:46445)
0812 05:24:17.333587 (+ 332us) server_negotiation.cc:206] Beginning negotiation
0812 05:24:17.333591 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 05:24:17.333727 (+ 136us) server_negotiation.cc:407] Connection header received
0812 05:24:17.333763 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:24:17.333765 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:24:17.333778 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:24:17.333842 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 05:24:17.334020 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:24:17.334792 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:24:17.336226 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:24:17.336920 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:24:17.336986 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:24:17.337086 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0812 05:24:17.337457 (+ 371us) server_negotiation.cc:299] Negotiation successful
0812 05:24:17.337499 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":303,"thread_start_us":64,"threads_started":1}
I20260812 05:24:17.337598 32190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:24:17.333120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.2:37537 (local address 127.0.0.1:40294)
0812 05:24:17.333306 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0812 05:24:17.333344 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0812 05:24:17.333409 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:24:17.333698 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:24:17.333701 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:24:17.333704 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 05:24:17.333778 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:24:17.333781 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:24:17.335061 (+ 1280us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:24:17.335062 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:24:17.336149 (+ 1087us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:24:17.336151 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:24:17.337322 (+ 1171us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:24:17.337323 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:24:17.337389 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:24:17.337394 (+ 5us) client_negotiation.cc:769] Sending connection context
0812 05:24:17.337470 (+ 76us) client_negotiation.cc:240] Negotiation successful
0812 05:24:17.337525 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"mutex_wait_us":30,"thread_start_us":113,"threads_started":1}
I20260812 05:24:17.337682 31619 tablet_service.cc:1467] Processing CreateTablet for tablet 8083a2c9a0ed473dbdde7b663e36a9bf (DEFAULT_TABLE table=test-table [id=ce25fffc23ee464e9231ccb1a38ec1cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 05:24:17.338011 32193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:24:17.333295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40294 (local address 127.31.100.2:37537)
0812 05:24:17.333519 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0812 05:24:17.333524 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 05:24:17.333536 (+ 12us) server_negotiation.cc:407] Connection header received
0812 05:24:17.333580 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:24:17.333582 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:24:17.333598 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:24:17.333690 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 05:24:17.333855 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:24:17.334973 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:24:17.336230 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:24:17.336926 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:24:17.337304 (+ 378us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:24:17.337583 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0812 05:24:17.337902 (+ 319us) server_negotiation.cc:299] Negotiation successful
0812 05:24:17.337947 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":186,"thread_start_us":86,"threads_started":1}
I20260812 05:24:17.338022 31619 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8083a2c9a0ed473dbdde7b663e36a9bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 05:24:17.338653 31845 tablet_service.cc:1467] Processing CreateTablet for tablet 8083a2c9a0ed473dbdde7b663e36a9bf (DEFAULT_TABLE table=test-table [id=ce25fffc23ee464e9231ccb1a38ec1cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 05:24:17.338955 31845 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8083a2c9a0ed473dbdde7b663e36a9bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 05:24:17.340740 32198 tablet_bootstrap.cc:492] T 8083a2c9a0ed473dbdde7b663e36a9bf P 73dd8c812ac3452a9f9a8822fc64620e: Bootstrap starting.
I20260812 05:24:17.341398 32199 tablet_bootstrap.cc:492] T 8083a2c9a0ed473dbdde7b663e36a9bf P 52d9d219ab104abca8578a3a296c8f51: Bootstrap starting.
I20260812 05:24:17.341626 32067 tablet_service.cc:1467] Processing CreateTablet for tablet 8083a2c9a0ed473dbdde7b663e36a9bf (DEFAULT_TABLE table=test-table [id=ce25fffc23ee464e9231ccb1a38ec1cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 05:24:17.341948 32198 tablet_bootstrap.cc:654] T 8083a2c9a0ed473dbdde7b663e36a9bf P 73dd8c812ac3452a9f9a8822fc64620e: Neither blocks nor log segments found. Creating new log.
I20260812 05:24:17.341996 32067 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8083a2c9a0ed473dbdde7b663e36a9bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 05:24:17.342070 32199 tablet_bootstrap.cc:654] T 8083a2c9a0ed473dbdde7b663e36a9bf P 52d9d219ab104abca8578a3a296c8f51: Neither blocks nor log segments found. Creating new log.
I20260812 05:24:17.342244 32198 log.cc:824] T 8083a2c9a0ed473dbdde7b663e36a9bf P 73dd8c812ac3452a9f9a8822fc64620e: Log is configured to *not* fsync() on all Append() calls
I20260812 05:24:17.342373 32199 log.cc:824] T 8083a2c9a0ed473dbdde7b663e36a9bf P 52d9d219ab104abca8578a3a296c8f51: Log is configured to *not* fsync() on all Append() calls
I20260812 05:24:17.343012 32199 tablet_bootstrap.cc:492] T 8083a2c9a0ed473dbdde7b663e36a9bf P 52d9d219ab104abca8578a3a296c8f51: No bootstrap required, opened a new log
I20260812 05:24:17.343053 32198 tablet_bootstrap.cc:492] T 8083a2c9a0ed473dbdde7b663e36a9bf P 73dd8c812ac3452a9f9a8822fc64620e: No bootstrap required, opened a new log
I20260812 05:24:17.343080 32199 ts_tablet_manager.cc:1397] T 8083a2c9a0ed473dbdde7b663e36a9bf P 52d9d219ab104abca8578a3a296c8f51: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260812 05:24:17.343117 32198 ts_tablet_manager.cc:1397] T 8083a2c9a0ed473dbdde7b663e36a9bf P 73dd8c812ac3452a9f9a8822fc64620e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260812 05:24:17.344764 32199 raft_consensus.cc:348] T 8083a2c9a0ed473dbdde7b663e36a9bf P 52d9d219ab104abca8578a3a296c8f51 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52d9d219ab104abca8578a3a296c8f51" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 37537 } } peers { permanent_uuid: "725e2e2910cd4f1098e084c017ac2aeb" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 46445 } } peers { permanent_uuid: "73dd8c812ac3452a9f9a8822fc64620e" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 44891 } }
I20260812 05:24:17.344892 32199 raft_consensus.cc:374] T 8083a2c9a0ed473dbdde7b663e36a9bf P 52d9d219ab104abca8578a3a296c8f51 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 05:24:17.344924 32199 raft_consensus.cc:729] T 8083a2c9a0ed473dbdde7b663e36a9bf P 52d9d219ab104abca8578a3a296c8f51 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52d9d219ab104abca8578a3a296c8f51, State: Initialized, Role: FOLLOWER
I20260812 05:24:17.345063 32199 consensus_queue.cc:260] T 8083a2c9a0ed473dbdde7b663e36a9bf P 52d9d219ab104abca8578a3a296c8f51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52d9d219ab104abca8578a3a296c8f51" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 37537 } } peers { permanent_uuid: "725e2e2910cd4f1098e084c017ac2aeb" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 46445 } } peers { permanent_uuid: "73dd8c812ac3452a9f9a8822fc64620e" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 44891 } }
I20260812 05:24:17.345322 32199 ts_tablet_manager.cc:1428] T 8083a2c9a0ed473dbdde7b663e36a9bf P 52d9d219ab104abca8578a3a296c8f51: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260812 05:24:17.345484 31953 heartbeater.cc:503] Master 127.31.100.62:43985 was elected leader, sending a full tablet report...
I20260812 05:24:17.345567 32207 tablet_bootstrap.cc:492] T 8083a2c9a0ed473dbdde7b663e36a9bf P 725e2e2910cd4f1098e084c017ac2aeb: Bootstrap starting.
I20260812 05:24:17.345544 32198 raft_consensus.cc:348] T 8083a2c9a0ed473dbdde7b663e36a9bf P 73dd8c812ac3452a9f9a8822fc64620e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52d9d219ab104abca8578a3a296c8f51" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 37537 } } peers { permanent_uuid: "725e2e2910cd4f1098e084c017ac2aeb" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 46445 } } peers { permanent_uuid: "73dd8c812ac3452a9f9a8822fc64620e" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 44891 } }
I20260812 05:24:17.345702 32198 raft_consensus.cc:374] T 8083a2c9a0ed473dbdde7b663e36a9bf P 73dd8c812ac3452a9f9a8822fc64620e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 05:24:17.345747 32198 raft_consensus.cc:729] T 8083a2c9a0ed473dbdde7b663e36a9bf P 73dd8c812ac3452a9f9a8822fc64620e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73dd8c812ac3452a9f9a8822fc64620e, State: Initialized, Role: FOLLOWER
I20260812 05:24:17.345880 32198 consensus_queue.cc:260] T 8083a2c9a0ed473dbdde7b663e36a9bf P 73dd8c812ac3452a9f9a8822fc64620e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52d9d219ab104abca8578a3a296c8f51" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 37537 } } peers { permanent_uuid: "725e2e2910cd4f1098e084c017ac2aeb" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 46445 } } peers { permanent_uuid: "73dd8c812ac3452a9f9a8822fc64620e" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 44891 } }
I20260812 05:24:17.346146 32198 ts_tablet_manager.cc:1428] T 8083a2c9a0ed473dbdde7b663e36a9bf P 73dd8c812ac3452a9f9a8822fc64620e: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260812 05:24:17.346179 32207 tablet_bootstrap.cc:654] T 8083a2c9a0ed473dbdde7b663e36a9bf P 725e2e2910cd4f1098e084c017ac2aeb: Neither blocks nor log segments found. Creating new log.
I20260812 05:24:17.346252 31728 heartbeater.cc:503] Master 127.31.100.62:43985 was elected leader, sending a full tablet report...
I20260812 05:24:17.346467 32207 log.cc:824] T 8083a2c9a0ed473dbdde7b663e36a9bf P 725e2e2910cd4f1098e084c017ac2aeb: Log is configured to *not* fsync() on all Append() calls
I20260812 05:24:17.347208 32207 tablet_bootstrap.cc:492] T 8083a2c9a0ed473dbdde7b663e36a9bf P 725e2e2910cd4f1098e084c017ac2aeb: No bootstrap required, opened a new log
I20260812 05:24:17.347262 32207 ts_tablet_manager.cc:1397] T 8083a2c9a0ed473dbdde7b663e36a9bf P 725e2e2910cd4f1098e084c017ac2aeb: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260812 05:24:17.348930 32207 raft_consensus.cc:348] T 8083a2c9a0ed473dbdde7b663e36a9bf P 725e2e2910cd4f1098e084c017ac2aeb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52d9d219ab104abca8578a3a296c8f51" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 37537 } } peers { permanent_uuid: "725e2e2910cd4f1098e084c017ac2aeb" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 46445 } } peers { permanent_uuid: "73dd8c812ac3452a9f9a8822fc64620e" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 44891 } }
I20260812 05:24:17.349079 32207 raft_consensus.cc:374] T 8083a2c9a0ed473dbdde7b663e36a9bf P 725e2e2910cd4f1098e084c017ac2aeb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 05:24:17.349131 32207 raft_consensus.cc:729] T 8083a2c9a0ed473dbdde7b663e36a9bf P 725e2e2910cd4f1098e084c017ac2aeb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 725e2e2910cd4f1098e084c017ac2aeb, State: Initialized, Role: FOLLOWER
I20260812 05:24:17.349241 32207 consensus_queue.cc:260] T 8083a2c9a0ed473dbdde7b663e36a9bf P 725e2e2910cd4f1098e084c017ac2aeb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52d9d219ab104abca8578a3a296c8f51" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 37537 } } peers { permanent_uuid: "725e2e2910cd4f1098e084c017ac2aeb" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 46445 } } peers { permanent_uuid: "73dd8c812ac3452a9f9a8822fc64620e" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 44891 } }
I20260812 05:24:17.349494 32207 ts_tablet_manager.cc:1428] T 8083a2c9a0ed473dbdde7b663e36a9bf P 725e2e2910cd4f1098e084c017ac2aeb: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260812 05:24:17.349550 32177 heartbeater.cc:503] Master 127.31.100.62:43985 was elected leader, sending a full tablet report...
I20260812 05:24:17.362475 32218 raft_consensus.cc:482] T 8083a2c9a0ed473dbdde7b663e36a9bf P 725e2e2910cd4f1098e084c017ac2aeb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 05:24:17.362552 32218 raft_consensus.cc:504] T 8083a2c9a0ed473dbdde7b663e36a9bf P 725e2e2910cd4f1098e084c017ac2aeb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52d9d219ab104abca8578a3a296c8f51" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 37537 } } peers { permanent_uuid: "725e2e2910cd4f1098e084c017ac2aeb" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 46445 } } peers { permanent_uuid: "73dd8c812ac3452a9f9a8822fc64620e" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 44891 } }
I20260812 05:24:17.362825 32218 leader_election.cc:288] T 8083a2c9a0ed473dbdde7b663e36a9bf P 725e2e2910cd4f1098e084c017ac2aeb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 52d9d219ab104abca8578a3a296c8f51 (127.31.100.2:37537), 73dd8c812ac3452a9f9a8822fc64620e (127.31.100.1:44891)
I20260812 05:24:17.366286 32197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:24:17.363012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.100.3:60383 (local address 127.31.100.1:44891)
0812 05:24:17.363065 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0812 05:24:17.363067 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 05:24:17.363176 (+ 109us) server_negotiation.cc:407] Connection header received
0812 05:24:17.363216 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:24:17.363217 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:24:17.363226 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:24:17.363271 (+ 45us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 05:24:17.363415 (+ 144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:24:17.364088 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:24:17.365383 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:24:17.365946 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:24:17.366020 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:24:17.366027 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0812 05:24:17.366191 (+ 164us) server_negotiation.cc:299] Negotiation successful
0812 05:24:17.366224 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260812 05:24:17.366286 32193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:24:17.362996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.100.3:43651 (local address 127.31.100.2:37537)
0812 05:24:17.363032 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0812 05:24:17.363033 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0812 05:24:17.363041 (+ 8us) server_negotiation.cc:407] Connection header received
0812 05:24:17.363117 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:24:17.363118 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:24:17.363126 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:24:17.363179 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 05:24:17.363308 (+ 129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:24:17.364039 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:24:17.365377 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:24:17.365946 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:24:17.366020 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:24:17.366027 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0812 05:24:17.366185 (+ 158us) server_negotiation.cc:299] Negotiation successful
0812 05:24:17.366223 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260812 05:24:17.366298 32179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:24:17.362957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.2:37537 (local address 127.31.100.3:43651)
0812 05:24:17.363005 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0812 05:24:17.363015 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0812 05:24:17.363051 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:24:17.363189 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:24:17.363190 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:24:17.363191 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 05:24:17.363251 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:24:17.363252 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:24:17.364110 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:24:17.364110 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:24:17.365298 (+ 1188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:24:17.365301 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:24:17.366020 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:24:17.366021 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:24:17.366118 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:24:17.366120 (+ 2us) client_negotiation.cc:769] Sending connection context
0812 05:24:17.366173 (+ 53us) client_negotiation.cc:240] Negotiation successful
0812 05:24:17.366204 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32,"mutex_wait_us":7}
I20260812 05:24:17.366312 32221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:24:17.362958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.1:44891 (local address 127.31.100.3:60383)
0812 05:24:17.363107 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0812 05:24:17.363116 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0812 05:24:17.363174 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:24:17.363291 (+ 117us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:24:17.363292 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:24:17.363293 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 05:24:17.363353 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:24:17.363353 (+ 0us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:24:17.364147 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:24:17.364148 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:24:17.365303 (+ 1155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:24:17.365304 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:24:17.366030 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:24:17.366031 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:24:17.366114 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:24:17.366117 (+ 3us) client_negotiation.cc:769] Sending connection context
0812 05:24:17.366167 (+ 50us) client_negotiation.cc:240] Negotiation successful
0812 05:24:17.366203 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"mutex_wait_us":34,"thread_start_us":84,"threads_started":1}
I20260812 05:24:17.366662 31874 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8083a2c9a0ed473dbdde7b663e36a9bf" candidate_uuid: "725e2e2910cd4f1098e084c017ac2aeb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52d9d219ab104abca8578a3a296c8f51" is_pre_election: true
I20260812 05:24:17.366781 31874 raft_consensus.cc:2454] T 8083a2c9a0ed473dbdde7b663e36a9bf P 52d9d219ab104abca8578a3a296c8f51 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 725e2e2910cd4f1098e084c017ac2aeb in term 0.
I20260812 05:24:17.366931 31656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8083a2c9a0ed473dbdde7b663e36a9bf" candidate_uuid: "725e2e2910cd4f1098e084c017ac2aeb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73dd8c812ac3452a9f9a8822fc64620e" is_pre_election: true
I20260812 05:24:17.366984 31979 leader_election.cc:302] T 8083a2c9a0ed473dbdde7b663e36a9bf P 725e2e2910cd4f1098e084c017ac2aeb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52d9d219ab104abca8578a3a296c8f51, 725e2e2910cd4f1098e084c017ac2aeb; no voters:
I20260812 05:24:17.367043 31656 raft_consensus.cc:2454] T 8083a2c9a0ed473dbdde7b663e36a9bf P 73dd8c812ac3452a9f9a8822fc64620e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 725e2e2910cd4f1098e084c017ac2aeb in term 0.
I20260812 05:24:17.367100 32218 raft_consensus.cc:2781] T 8083a2c9a0ed473dbdde7b663e36a9bf P 725e2e2910cd4f1098e084c017ac2aeb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 05:24:17.367158 32218 raft_consensus.cc:482] T 8083a2c9a0ed473dbdde7b663e36a9bf P 725e2e2910cd4f1098e084c017ac2aeb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 05:24:17.367182 32218 raft_consensus.cc:3037] T 8083a2c9a0ed473dbdde7b663e36a9bf P 725e2e2910cd4f1098e084c017ac2aeb [term 0 FOLLOWER]: Advancing to term 1
I20260812 05:24:17.367764 32218 raft_consensus.cc:504] T 8083a2c9a0ed473dbdde7b663e36a9bf P 725e2e2910cd4f1098e084c017ac2aeb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52d9d219ab104abca8578a3a296c8f51" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 37537 } } peers { permanent_uuid: "725e2e2910cd4f1098e084c017ac2aeb" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 46445 } } peers { permanent_uuid: "73dd8c812ac3452a9f9a8822fc64620e" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 44891 } }
I20260812 05:24:17.367873 32218 leader_election.cc:288] T 8083a2c9a0ed473dbdde7b663e36a9bf P 725e2e2910cd4f1098e084c017ac2aeb [CANDIDATE]: Term 1 election: Requested vote from peers 52d9d219ab104abca8578a3a296c8f51 (127.31.100.2:37537), 73dd8c812ac3452a9f9a8822fc64620e (127.31.100.1:44891)
I20260812 05:24:17.368031 31874 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8083a2c9a0ed473dbdde7b663e36a9bf" candidate_uuid: "725e2e2910cd4f1098e084c017ac2aeb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52d9d219ab104abca8578a3a296c8f51"
I20260812 05:24:17.368039 31656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8083a2c9a0ed473dbdde7b663e36a9bf" candidate_uuid: "725e2e2910cd4f1098e084c017ac2aeb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73dd8c812ac3452a9f9a8822fc64620e"
I20260812 05:24:17.368098 31874 raft_consensus.cc:3037] T 8083a2c9a0ed473dbdde7b663e36a9bf P 52d9d219ab104abca8578a3a296c8f51 [term 0 FOLLOWER]: Advancing to term 1
I20260812 05:24:17.368111 31656 raft_consensus.cc:3037] T 8083a2c9a0ed473dbdde7b663e36a9bf P 73dd8c812ac3452a9f9a8822fc64620e [term 0 FOLLOWER]: Advancing to term 1
I20260812 05:24:17.368737 31874 raft_consensus.cc:2454] T 8083a2c9a0ed473dbdde7b663e36a9bf P 52d9d219ab104abca8578a3a296c8f51 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 725e2e2910cd4f1098e084c017ac2aeb in term 1.
I20260812 05:24:17.368748 31656 raft_consensus.cc:2454] T 8083a2c9a0ed473dbdde7b663e36a9bf P 73dd8c812ac3452a9f9a8822fc64620e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 725e2e2910cd4f1098e084c017ac2aeb in term 1.
I20260812 05:24:17.368898 31978 leader_election.cc:302] T 8083a2c9a0ed473dbdde7b663e36a9bf P 725e2e2910cd4f1098e084c017ac2aeb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 725e2e2910cd4f1098e084c017ac2aeb, 73dd8c812ac3452a9f9a8822fc64620e; no voters:
I20260812 05:24:17.368997 32218 raft_consensus.cc:2781] T 8083a2c9a0ed473dbdde7b663e36a9bf P 725e2e2910cd4f1098e084c017ac2aeb [term 1 FOLLOWER]: Leader election won for term 1
I20260812 05:24:17.369143 32218 raft_consensus.cc:686] T 8083a2c9a0ed473dbdde7b663e36a9bf P 725e2e2910cd4f1098e084c017ac2aeb [term 1 LEADER]: Becoming Leader. State: Replica: 725e2e2910cd4f1098e084c017ac2aeb, State: Running, Role: LEADER
I20260812 05:24:17.369246 32218 consensus_queue.cc:237] T 8083a2c9a0ed473dbdde7b663e36a9bf P 725e2e2910cd4f1098e084c017ac2aeb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52d9d219ab104abca8578a3a296c8f51" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 37537 } } peers { permanent_uuid: "725e2e2910cd4f1098e084c017ac2aeb" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 46445 } } peers { permanent_uuid: "73dd8c812ac3452a9f9a8822fc64620e" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 44891 } }
I20260812 05:24:17.369995 31406 catalog_manager.cc:5495] T 8083a2c9a0ed473dbdde7b663e36a9bf P 725e2e2910cd4f1098e084c017ac2aeb reported cstate change: term changed from 0 to 1, leader changed from <none> to 725e2e2910cd4f1098e084c017ac2aeb (127.31.100.3). New cstate: current_term: 1 leader_uuid: "725e2e2910cd4f1098e084c017ac2aeb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52d9d219ab104abca8578a3a296c8f51" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 37537 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "725e2e2910cd4f1098e084c017ac2aeb" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 46445 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "73dd8c812ac3452a9f9a8822fc64620e" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 44891 } health_report { overall_health: UNKNOWN } } }
I20260812 05:24:17.376119 32192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:24:17.374248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34654 (local address 127.31.100.3:46445)
0812 05:24:17.374284 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0812 05:24:17.374286 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 05:24:17.374294 (+ 8us) server_negotiation.cc:407] Connection header received
0812 05:24:17.374348 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:24:17.374349 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:24:17.374365 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:24:17.374415 (+ 50us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 05:24:17.374508 (+ 93us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:24:17.374924 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:24:17.375562 (+ 638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:24:17.375730 (+ 168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:24:17.375767 (+ 37us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:24:17.375880 (+ 113us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 05:24:17.375952 (+ 72us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 05:24:17.376011 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0812 05:24:17.376052 (+ 41us) server_negotiation.cc:299] Negotiation successful
0812 05:24:17.376058 (+ 6us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260812 05:24:17.377605 31656 raft_consensus.cc:1261] T 8083a2c9a0ed473dbdde7b663e36a9bf P 73dd8c812ac3452a9f9a8822fc64620e [term 1 FOLLOWER]: Refusing update from remote peer 725e2e2910cd4f1098e084c017ac2aeb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 05:24:17.377607 31874 raft_consensus.cc:1261] T 8083a2c9a0ed473dbdde7b663e36a9bf P 52d9d219ab104abca8578a3a296c8f51 [term 1 FOLLOWER]: Refusing update from remote peer 725e2e2910cd4f1098e084c017ac2aeb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 05:24:17.377907 32223 consensus_queue.cc:1035] T 8083a2c9a0ed473dbdde7b663e36a9bf P 725e2e2910cd4f1098e084c017ac2aeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "73dd8c812ac3452a9f9a8822fc64620e" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 44891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 05:24:17.378017 32218 consensus_queue.cc:1035] T 8083a2c9a0ed473dbdde7b663e36a9bf P 725e2e2910cd4f1098e084c017ac2aeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "52d9d219ab104abca8578a3a296c8f51" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 37537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 05:24:17.380026 32230 mvcc.cc:205] Tried to move back new op lower bound from 7317554206216413184 to 7317554206185066496. Current Snapshot: MvccSnapshot[applied={T|T < 7317554206216413184}]
I20260812 05:24:17.380316 32231 mvcc.cc:205] Tried to move back new op lower bound from 7317554206216413184 to 7317554206185066496. Current Snapshot: MvccSnapshot[applied={T|T < 7317554206216413184}]
I20260812 05:24:17.380658 32233 mvcc.cc:205] Tried to move back new op lower bound from 7317554206216413184 to 7317554206185066496. Current Snapshot: MvccSnapshot[applied={T|T < 7317554206216413184}]
I20260812 05:24:17.382815 31406 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39556:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 05:24:17.382905 31406 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39556:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 05:24:17.383535 31406 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 67ce6d1cc1784329a55ccc230853942d: Sending DeleteTablet for 3 replicas of tablet 8083a2c9a0ed473dbdde7b663e36a9bf
I20260812 05:24:17.383857 31619 tablet_service.cc:1514] Processing DeleteTablet for tablet 8083a2c9a0ed473dbdde7b663e36a9bf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 05:24:17 UTC) from {username='slave'} at 127.0.0.1:48440
I20260812 05:24:17.383900 31845 tablet_service.cc:1514] Processing DeleteTablet for tablet 8083a2c9a0ed473dbdde7b663e36a9bf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 05:24:17 UTC) from {username='slave'} at 127.0.0.1:40294
I20260812 05:24:17.384070 32247 tablet_replica.cc:330] T 8083a2c9a0ed473dbdde7b663e36a9bf P 52d9d219ab104abca8578a3a296c8f51: stopping tablet replica
I20260812 05:24:17.384160 32247 raft_consensus.cc:2229] T 8083a2c9a0ed473dbdde7b663e36a9bf P 52d9d219ab104abca8578a3a296c8f51 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 05:24:17.384169 32144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0waFvT/build/release/bin/kudu with pid 31518
I20260812 05:24:17.384249 32247 raft_consensus.cc:2258] T 8083a2c9a0ed473dbdde7b663e36a9bf P 52d9d219ab104abca8578a3a296c8f51 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 05:24:17.384356 32067 tablet_service.cc:1514] Processing DeleteTablet for tablet 8083a2c9a0ed473dbdde7b663e36a9bf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 05:24:17 UTC) from {username='slave'} at 127.0.0.1:34646
I20260812 05:24:17.384765 32247 ts_tablet_manager.cc:1905] T 8083a2c9a0ed473dbdde7b663e36a9bf P 52d9d219ab104abca8578a3a296c8f51: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 05:24:17.387457 32247 ts_tablet_manager.cc:1918] T 8083a2c9a0ed473dbdde7b663e36a9bf P 52d9d219ab104abca8578a3a296c8f51: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 05:24:17.387539 32247 log.cc:1196] T 8083a2c9a0ed473dbdde7b663e36a9bf P 52d9d219ab104abca8578a3a296c8f51: Deleting WAL directory at /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786512017613366-32144-0/minicluster-data/ts-1/wal/wals/8083a2c9a0ed473dbdde7b663e36a9bf
I20260812 05:24:17.387750 32247 ts_tablet_manager.cc:1939] T 8083a2c9a0ed473dbdde7b663e36a9bf P 52d9d219ab104abca8578a3a296c8f51: Deleting consensus metadata
I20260812 05:24:17.388085 32250 tablet_replica.cc:330] T 8083a2c9a0ed473dbdde7b663e36a9bf P 725e2e2910cd4f1098e084c017ac2aeb: stopping tablet replica
I20260812 05:24:17.388185 32250 raft_consensus.cc:2229] T 8083a2c9a0ed473dbdde7b663e36a9bf P 725e2e2910cd4f1098e084c017ac2aeb [term 1 LEADER]: Raft consensus shutting down.
I20260812 05:24:17.388300 32250 raft_consensus.cc:2258] T 8083a2c9a0ed473dbdde7b663e36a9bf P 725e2e2910cd4f1098e084c017ac2aeb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 05:24:17.388778 32250 ts_tablet_manager.cc:1905] T 8083a2c9a0ed473dbdde7b663e36a9bf P 725e2e2910cd4f1098e084c017ac2aeb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 05:24:17.389973 31384 catalog_manager.cc:4841] TS 52d9d219ab104abca8578a3a296c8f51 (127.31.100.2:37537): tablet 8083a2c9a0ed473dbdde7b663e36a9bf (table test-table [id=ce25fffc23ee464e9231ccb1a38ec1cb]) successfully deleted
I20260812 05:24:17.390378 32250 ts_tablet_manager.cc:1918] T 8083a2c9a0ed473dbdde7b663e36a9bf P 725e2e2910cd4f1098e084c017ac2aeb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 05:24:17.390437 32250 log.cc:1196] T 8083a2c9a0ed473dbdde7b663e36a9bf P 725e2e2910cd4f1098e084c017ac2aeb: Deleting WAL directory at /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786512017613366-32144-0/minicluster-data/ts-2/wal/wals/8083a2c9a0ed473dbdde7b663e36a9bf
I20260812 05:24:17.390656 32250 ts_tablet_manager.cc:1939] T 8083a2c9a0ed473dbdde7b663e36a9bf P 725e2e2910cd4f1098e084c017ac2aeb: Deleting consensus metadata
I20260812 05:24:17.391047 31382 catalog_manager.cc:4841] TS 725e2e2910cd4f1098e084c017ac2aeb (127.31.100.3:46445): tablet 8083a2c9a0ed473dbdde7b663e36a9bf (table test-table [id=ce25fffc23ee464e9231ccb1a38ec1cb]) successfully deleted
W20260812 05:24:17.393548 31383 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.31.100.1:44891 (error 108)
W20260812 05:24:17.394114 32190 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 05:24:17.393857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.1:44891 (local address 127.0.0.1:48452)
0812 05:24:17.393909 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0812 05:24:17.393921 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0812 05:24:17.393966 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:24:17.394066 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.100.1:44891: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":33}
W20260812 05:24:17.394284 31383 catalog_manager.cc:4584] TS 73dd8c812ac3452a9f9a8822fc64620e (127.31.100.1:44891): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8083a2c9a0ed473dbdde7b663e36a9bf: Network error: Client connection negotiation failed: client connection to 127.31.100.1:44891: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260812 05:24:17.394323 32144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0waFvT/build/release/bin/kudu with pid 31738
I20260812 05:24:17.409789 32144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0waFvT/build/release/bin/kudu with pid 31956
I20260812 05:24:17.415827 32144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0waFvT/build/release/bin/kudu with pid 31365
2026-08-12T05:24:17Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (656 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786512017613366-32144-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 05:24:17 dist-test-slave-42z9 krb5kdc[32257](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 05:24:17 dist-test-slave-42z9 krb5kdc[32257](info): set up 2 sockets
Aug 12 05:24:17 dist-test-slave-42z9 krb5kdc[32257](info): commencing operation
krb5kdc: starting...
W20260812 05:24:18.585783 32144 mini_kdc.cc:121] Time spent starting KDC: real 1.145s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786512017613366-32144-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786512017613366-32144-0/krb5kdc/test-user.keytab.
Aug 12 05:24:18 dist-test-slave-42z9 krb5kdc[32257](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512258, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T05:24:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T05:24:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.100.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.100.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.100.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.62.keytab.
Entry for principal kudu/127.31.100.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.62.keytab.
I20260812 05:24:18.699621 32144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0waFvT/build/release/bin/kudu
/tmp/dist-test-task0waFvT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786512017613366-32144-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786512017613366-32144-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786512017613366-32144-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786512017613366-32144-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.100.62:42887
--webserver_interface=127.31.100.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.62.keytab
--principal=kudu/127.31.100.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.100.20:35063
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.100.62:42887
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786512017613366-32144-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786512017613366-32144-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786512017613366-32144-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 05:24:18.780138 32275 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 05:24:18.780341 32275 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 05:24:18.780362 32275 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 05:24:18.781472 32275 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 05:24:18.781505 32275 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 05:24:18.781518 32275 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 05:24:18.781529 32275 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 05:24:18.781540 32275 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 05:24:18.782961 32275 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.100.20:35063
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786512017613366-32144-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786512017613366-32144-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.100.62:42887
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.62.keytab
--principal=kudu/127.31.100.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.100.62:42887
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786512017613366-32144-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.100.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32275
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786512017613366-32144-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 04:20:34 UTC on ccycloud.quasar-phqdqp.root.comops.site
build id 4581871
I20260812 05:24:18.783155 32275 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 05:24:18.783324 32275 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 05:24:18.787865 32281 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 05:24:18.787889 32282 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 05:24:18.788089 32284 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 05:24:18.788112 32275 server_base.cc:1034] running on GCE node
I20260812 05:24:18.788334 32275 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 05:24:18.788621 32275 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 05:24:18.789758 32275 hybrid_clock.cc:648] HybridClock initialized: now 1786512258789741 us; error 29 us; skew 500 ppm
Aug 12 05:24:18 dist-test-slave-42z9 krb5kdc[32257](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512258, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.100.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 05:24:18.792012 32275 init.cc:377] Logged in from keytab as kudu/127.31.100.62@KRBTEST.COM (short username kudu)
I20260812 05:24:18.792441 32275 webserver.cc:460] Webserver started at http://127.31.100.62:33185/ using document root <none> and password file <none>
I20260812 05:24:18.792662 32275 fs_manager.cc:359] Metadata directory not provided
I20260812 05:24:18.792712 32275 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 05:24:18.792838 32275 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 05:24:18.793883 32275 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786512017613366-32144-0/minicluster-data/master-0/data/instance:
uuid: "77445fd1d2d04cbfb61323303f539507"
format_stamp: "Formatted at 2026-08-12 05:24:18 on dist-test-slave-42z9"
server_key: "ce3a8a915f3d49327860cebe77157958"
server_key_iv: "fef7212374c17ecf69024c0ee031e6cf"
server_key_version: "encryptionkey@0"
I20260812 05:24:18.794206 32275 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786512017613366-32144-0/minicluster-data/master-0/wal/instance:
uuid: "77445fd1d2d04cbfb61323303f539507"
format_stamp: "Formatted at 2026-08-12 05:24:18 on dist-test-slave-42z9"
server_key: "ce3a8a915f3d49327860cebe77157958"
server_key_iv: "fef7212374c17ecf69024c0ee031e6cf"
server_key_version: "encryptionkey@0"
I20260812 05:24:18.795398 32275 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260812 05:24:18.796031 32297 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 05:24:18.796177 32275 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260812 05:24:18.796262 32275 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786512017613366-32144-0/minicluster-data/master-0/data,/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786512017613366-32144-0/minicluster-data/master-0/wal
uuid: "77445fd1d2d04cbfb61323303f539507"
format_stamp: "Formatted at 2026-08-12 05:24:18 on dist-test-slave-42z9"
server_key: "ce3a8a915f3d49327860cebe77157958"
server_key_iv: "fef7212374c17ecf69024c0ee031e6cf"
server_key_version: "encryptionkey@0"
I20260812 05:24:18.796319 32275 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786512017613366-32144-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786512017613366-32144-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786512017613366-32144-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 05:24:18.805292 32275 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 05:24:18.806921 32275 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 05:24:18.807046 32275 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 05:24:18.810956 32275 rpc_server.cc:307] RPC server started. Bound to: 127.31.100.62:42887
I20260812 05:24:18.810988 32382 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.100.62:42887 every 8 connection(s)
I20260812 05:24:18.811285 32275 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786512017613366-32144-0/minicluster-data/master-0/data/info.pb
I20260812 05:24:18.812079 32384 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 05:24:18.814028 32144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0waFvT/build/release/bin/kudu as pid 32275
I20260812 05:24:18.814134 32144 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786512017613366-32144-0/minicluster-data/master-0/wal/instance
I20260812 05:24:18.814294 32144 external_mini_cluster.cc:1428] Setting key e410a0bb75176318524ae4945d3f5372
I20260812 05:24:18.814308 32384 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 77445fd1d2d04cbfb61323303f539507: Bootstrap starting.
I20260812 05:24:18.814925 32384 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 77445fd1d2d04cbfb61323303f539507: Neither blocks nor log segments found. Creating new log.
I20260812 05:24:18.815220 32384 log.cc:824] T 00000000000000000000000000000000 P 77445fd1d2d04cbfb61323303f539507: Log is configured to *not* fsync() on all Append() calls
I20260812 05:24:18.815894 32384 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 77445fd1d2d04cbfb61323303f539507: No bootstrap required, opened a new log
I20260812 05:24:18.818162 32384 raft_consensus.cc:348] T 00000000000000000000000000000000 P 77445fd1d2d04cbfb61323303f539507 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77445fd1d2d04cbfb61323303f539507" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 42887 } }
I20260812 05:24:18.818307 32384 raft_consensus.cc:374] T 00000000000000000000000000000000 P 77445fd1d2d04cbfb61323303f539507 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 05:24:18.818344 32384 raft_consensus.cc:729] T 00000000000000000000000000000000 P 77445fd1d2d04cbfb61323303f539507 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77445fd1d2d04cbfb61323303f539507, State: Initialized, Role: FOLLOWER
I20260812 05:24:18.818462 32384 consensus_queue.cc:260] T 00000000000000000000000000000000 P 77445fd1d2d04cbfb61323303f539507 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77445fd1d2d04cbfb61323303f539507" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 42887 } }
I20260812 05:24:18.818531 32384 raft_consensus.cc:388] T 00000000000000000000000000000000 P 77445fd1d2d04cbfb61323303f539507 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 05:24:18.818565 32384 raft_consensus.cc:482] T 00000000000000000000000000000000 P 77445fd1d2d04cbfb61323303f539507 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 05:24:18.818617 32384 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 77445fd1d2d04cbfb61323303f539507 [term 0 FOLLOWER]: Advancing to term 1
I20260812 05:24:18.819252 32384 raft_consensus.cc:504] T 00000000000000000000000000000000 P 77445fd1d2d04cbfb61323303f539507 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77445fd1d2d04cbfb61323303f539507" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 42887 } }
I20260812 05:24:18.819353 32384 leader_election.cc:302] T 00000000000000000000000000000000 P 77445fd1d2d04cbfb61323303f539507 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 77445fd1d2d04cbfb61323303f539507; no voters:
I20260812 05:24:18.819537 32384 leader_election.cc:288] T 00000000000000000000000000000000 P 77445fd1d2d04cbfb61323303f539507 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 05:24:18.819567 32393 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 77445fd1d2d04cbfb61323303f539507 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 05:24:18.819705 32393 raft_consensus.cc:686] T 00000000000000000000000000000000 P 77445fd1d2d04cbfb61323303f539507 [term 1 LEADER]: Becoming Leader. State: Replica: 77445fd1d2d04cbfb61323303f539507, State: Running, Role: LEADER
I20260812 05:24:18.819809 32384 sys_catalog.cc:564] T 00000000000000000000000000000000 P 77445fd1d2d04cbfb61323303f539507 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 05:24:18.819818 32393 consensus_queue.cc:237] T 00000000000000000000000000000000 P 77445fd1d2d04cbfb61323303f539507 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77445fd1d2d04cbfb61323303f539507" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 42887 } }
I20260812 05:24:18.820181 32394 sys_catalog.cc:455] T 00000000000000000000000000000000 P 77445fd1d2d04cbfb61323303f539507 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "77445fd1d2d04cbfb61323303f539507" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77445fd1d2d04cbfb61323303f539507" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 42887 } } }
I20260812 05:24:18.820258 32394 sys_catalog.cc:458] T 00000000000000000000000000000000 P 77445fd1d2d04cbfb61323303f539507 [sys.catalog]: This master's current role is: LEADER
I20260812 05:24:18.820411 32395 sys_catalog.cc:455] T 00000000000000000000000000000000 P 77445fd1d2d04cbfb61323303f539507 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 77445fd1d2d04cbfb61323303f539507. Latest consensus state: current_term: 1 leader_uuid: "77445fd1d2d04cbfb61323303f539507" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77445fd1d2d04cbfb61323303f539507" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 42887 } } }
I20260812 05:24:18.820472 32395 sys_catalog.cc:458] T 00000000000000000000000000000000 P 77445fd1d2d04cbfb61323303f539507 [sys.catalog]: This master's current role is: LEADER
I20260812 05:24:18.820596 32402 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 05:24:18.821063 32402 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Aug 12 05:24:18 dist-test-slave-42z9 krb5kdc[32257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512258, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.100.62@KRBTEST.COM
I20260812 05:24:18.822571 32402 catalog_manager.cc:1347] Generated new cluster ID: da6a5ae714a44f6abfe7e31c8615fbb7
I20260812 05:24:18.822633 32402 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 05:24:18.825256 32390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:24:18.814764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40792 (local address 127.31.100.62:42887)
0812 05:24:18.814924 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0812 05:24:18.814931 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 05:24:18.814958 (+ 27us) server_negotiation.cc:407] Connection header received
0812 05:24:18.815413 (+ 455us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:24:18.815420 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:24:18.815467 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:24:18.815575 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:24:18.816196 (+ 621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:24:18.817563 (+ 1367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:24:18.818283 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:24:18.818865 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:24:18.818986 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:24:18.821684 (+ 2698us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 05:24:18.821688 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 05:24:18.821688 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 05:24:18.821699 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0812 05:24:18.823705 (+ 2006us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:24:18.824079 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:24:18.824082 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:24:18.824082 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:24:18.824186 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:24:18.824362 (+ 176us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:24:18.824363 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:24:18.824364 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:24:18.824663 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 05:24:18.824815 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0812 05:24:18.824918 (+ 103us) server_negotiation.cc:299] Negotiation successful
0812 05:24:18.825050 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":62,"threads_started":1}
I20260812 05:24:18.827052 32402 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 05:24:18.827715 32402 catalog_manager.cc:1504] Loading token signing keys...
I20260812 05:24:18.834481 32402 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 77445fd1d2d04cbfb61323303f539507: Generated new TSK 0
I20260812 05:24:18.834637 32402 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.31.100.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.100.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.100.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.1.keytab.
Entry for principal kudu/127.31.100.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.1.keytab.
I20260812 05:24:18.862390 32144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0waFvT/build/release/bin/kudu
/tmp/dist-test-task0waFvT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786512017613366-32144-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786512017613366-32144-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786512017613366-32144-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786512017613366-32144-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.100.1:0
--local_ip_for_outbound_sockets=127.31.100.1
--webserver_interface=127.31.100.1
--webserver_port=0
--tserver_master_addrs=127.31.100.62:42887
--keytab_file=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.1.keytab
--principal=kudu/127.31.100.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.100.20:35063
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786512017613366-32144-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786512017613366-32144-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786512017613366-32144-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 05:24:18.943179 32432 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 05:24:18.943373 32432 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 05:24:18.943398 32432 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 05:24:18.944533 32432 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 05:24:18.944573 32432 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 05:24:18.944598 32432 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.100.1
I20260812 05:24:18.946076 32432 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.100.20:35063
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786512017613366-32144-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786512017613366-32144-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.1.keytab
--principal=kudu/127.31.100.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.100.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786512017613366-32144-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.100.1
--webserver_port=0
--tserver_master_addrs=127.31.100.62:42887
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32432
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.100.1
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786512017613366-32144-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 04:20:34 UTC on ccycloud.quasar-phqdqp.root.comops.site
build id 4581871
I20260812 05:24:18.946280 32432 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 05:24:18.946480 32432 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 05:24:18.951197 32440 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 05:24:18.951197 32445 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 05:24:18.951304 32432 server_base.cc:1034] running on GCE node
W20260812 05:24:18.951422 32438 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 05:24:18.951618 32432 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 05:24:18.951830 32432 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 05:24:18.952965 32432 hybrid_clock.cc:648] HybridClock initialized: now 1786512258952952 us; error 29 us; skew 500 ppm
Aug 12 05:24:18 dist-test-slave-42z9 krb5kdc[32257](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512258, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.100.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 05:24:18.955417 32432 init.cc:377] Logged in from keytab as kudu/127.31.100.1@KRBTEST.COM (short username kudu)
I20260812 05:24:18.955845 32432 webserver.cc:460] Webserver started at http://127.31.100.1:33881/ using document root <none> and password file <none>
I20260812 05:24:18.956060 32432 fs_manager.cc:359] Metadata directory not provided
I20260812 05:24:18.956107 32432 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 05:24:18.956233 32432 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 05:24:18.957261 32432 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786512017613366-32144-0/minicluster-data/ts-0/data/instance:
uuid: "4b4c5f4c2c1d444b8550f241e84a182e"
format_stamp: "Formatted at 2026-08-12 05:24:18 on dist-test-slave-42z9"
server_key: "1dfdb8b8594148aad0a4d20ccbb3e01a"
server_key_iv: "8d679b3c44a5222cb246d80202964b30"
server_key_version: "encryptionkey@0"
I20260812 05:24:18.957617 32432 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786512017613366-32144-0/minicluster-data/ts-0/wal/instance:
uuid: "4b4c5f4c2c1d444b8550f241e84a182e"
format_stamp: "Formatted at 2026-08-12 05:24:18 on dist-test-slave-42z9"
server_key: "1dfdb8b8594148aad0a4d20ccbb3e01a"
server_key_iv: "8d679b3c44a5222cb246d80202964b30"
server_key_version: "encryptionkey@0"
I20260812 05:24:18.958940 32432 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260812 05:24:18.959663 32458 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 05:24:18.959812 32432 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260812 05:24:18.959898 32432 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786512017613366-32144-0/minicluster-data/ts-0/data,/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786512017613366-32144-0/minicluster-data/ts-0/wal
uuid: "4b4c5f4c2c1d444b8550f241e84a182e"
format_stamp: "Formatted at 2026-08-12 05:24:18 on dist-test-slave-42z9"
server_key: "1dfdb8b8594148aad0a4d20ccbb3e01a"
server_key_iv: "8d679b3c44a5222cb246d80202964b30"
server_key_version: "encryptionkey@0"
I20260812 05:24:18.959964 32432 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786512017613366-32144-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786512017613366-32144-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786512017613366-32144-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 05:24:18.972082 32432 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 05:24:18.974144 32432 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 05:24:18.974272 32432 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 05:24:18.974469 32432 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 05:24:18.974745 32432 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 05:24:18.974776 32432 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 05:24:18.974802 32432 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 05:24:18.974825 32432 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 05:24:18.980247 32432 rpc_server.cc:307] RPC server started. Bound to: 127.31.100.1:33067
I20260812 05:24:18.980291 32636 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.100.1:33067 every 8 connection(s)
I20260812 05:24:18.980585 32432 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786512017613366-32144-0/minicluster-data/ts-0/data/info.pb
Aug 12 05:24:18 dist-test-slave-42z9 krb5kdc[32257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512258, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.100.1@KRBTEST.COM for kudu/127.31.100.62@KRBTEST.COM
I20260812 05:24:18.986917 32144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0waFvT/build/release/bin/kudu as pid 32432
I20260812 05:24:18.987035 32144 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786512017613366-32144-0/minicluster-data/ts-0/wal/instance
I20260812 05:24:18.987224 32144 external_mini_cluster.cc:1428] Setting key 37d79292736b6280fa8ef826e199ca30
I20260812 05:24:18.989183 32390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:24:18.981253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.100.1:50745 (local address 127.31.100.62:42887)
0812 05:24:18.981297 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0812 05:24:18.981299 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 05:24:18.981491 (+ 192us) server_negotiation.cc:407] Connection header received
0812 05:24:18.981984 (+ 493us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:24:18.981985 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:24:18.981996 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:24:18.982053 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:24:18.982884 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:24:18.983399 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:24:18.984304 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:24:18.984541 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:24:18.984596 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:24:18.986686 (+ 2090us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 05:24:18.986688 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 05:24:18.986688 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 05:24:18.986695 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 05:24:18.987984 (+ 1289us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:24:18.988387 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:24:18.988388 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:24:18.988388 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:24:18.988435 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:24:18.988645 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:24:18.988645 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:24:18.988645 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:24:18.988736 (+ 91us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 05:24:18.988780 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0812 05:24:18.989052 (+ 272us) server_negotiation.cc:299] Negotiation successful
0812 05:24:18.989119 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260812 05:24:18.989405 32641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:24:18.981202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.62:42887 (local address 127.31.100.1:50745)
0812 05:24:18.981360 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0812 05:24:18.981399 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0812 05:24:18.981882 (+ 483us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:24:18.982103 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:24:18.982106 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:24:18.982639 (+ 533us) client_negotiation.cc:189] Negotiated authn=SASL
0812 05:24:18.982815 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:24:18.982819 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:24:18.983467 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:24:18.983468 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:24:18.984238 (+ 770us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:24:18.984240 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:24:18.984623 (+ 383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:24:18.984624 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:24:18.984726 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:24:18.985155 (+ 429us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 05:24:18.985163 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0812 05:24:18.986592 (+ 1429us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 05:24:18.988061 (+ 1469us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 05:24:18.988069 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 05:24:18.988071 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 05:24:18.988326 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 05:24:18.988471 (+ 145us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 05:24:18.988472 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 05:24:18.988472 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 05:24:18.988596 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 05:24:18.988783 (+ 187us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 05:24:18.988787 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 05:24:18.988927 (+ 140us) client_negotiation.cc:769] Sending connection context
0812 05:24:18.989033 (+ 106us) client_negotiation.cc:240] Negotiation successful
0812 05:24:18.989194 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":84,"threads_started":1}
I20260812 05:24:18.989866 32637 heartbeater.cc:347] Connected to a master server at 127.31.100.62:42887
I20260812 05:24:18.989950 32637 heartbeater.cc:464] Registering TS with master...
I20260812 05:24:18.990108 32637 heartbeater.cc:511] Master 127.31.100.62:42887 requested a full tablet report, sending...
I20260812 05:24:18.990613 32325 ts_manager.cc:194] Registered new tserver with Master: 4b4c5f4c2c1d444b8550f241e84a182e (127.31.100.1:33067)
I20260812 05:24:18.991437 32325 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.100.1@KRBTEST.COM'} at 127.31.100.1:50745
WARNING: no policy specified for kudu/127.31.100.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.100.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.100.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.2.keytab.
Entry for principal kudu/127.31.100.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.2.keytab.
I20260812 05:24:19.011606 32144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0waFvT/build/release/bin/kudu
/tmp/dist-test-task0waFvT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786512017613366-32144-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786512017613366-32144-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786512017613366-32144-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786512017613366-32144-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.100.2:0
--local_ip_for_outbound_sockets=127.31.100.2
--webserver_interface=127.31.100.2
--webserver_port=0
--tserver_master_addrs=127.31.100.62:42887
--keytab_file=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.2.keytab
--principal=kudu/127.31.100.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.100.20:35063
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786512017613366-32144-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786512017613366-32144-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786512017613366-32144-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 05:24:19.092612 32645 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 05:24:19.092808 32645 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 05:24:19.092830 32645 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 05:24:19.093946 32645 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 05:24:19.093981 32645 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 05:24:19.094004 32645 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.100.2
I20260812 05:24:19.095392 32645 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.100.20:35063
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786512017613366-32144-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786512017613366-32144-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.2.keytab
--principal=kudu/127.31.100.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.100.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786512017613366-32144-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.100.2
--webserver_port=0
--tserver_master_addrs=127.31.100.62:42887
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32645
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.100.2
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786512017613366-32144-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 04:20:34 UTC on ccycloud.quasar-phqdqp.root.comops.site
build id 4581871
I20260812 05:24:19.095575 32645 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 05:24:19.095788 32645 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 05:24:19.100757 32657 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 05:24:19.100772 32664 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 05:24:19.100767 32656 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 05:24:19.101095 32645 server_base.cc:1034] running on GCE node
I20260812 05:24:19.101254 32645 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 05:24:19.101457 32645 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 05:24:19.102596 32645 hybrid_clock.cc:648] HybridClock initialized: now 1786512259102588 us; error 28 us; skew 500 ppm
Aug 12 05:24:19 dist-test-slave-42z9 krb5kdc[32257](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512259, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.100.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 05:24:19.104945 32645 init.cc:377] Logged in from keytab as kudu/127.31.100.2@KRBTEST.COM (short username kudu)
I20260812 05:24:19.105353 32645 webserver.cc:460] Webserver started at http://127.31.100.2:34387/ using document root <none> and password file <none>
I20260812 05:24:19.105597 32645 fs_manager.cc:359] Metadata directory not provided
I20260812 05:24:19.105643 32645 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 05:24:19.105754 32645 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 05:24:19.106743 32645 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786512017613366-32144-0/minicluster-data/ts-1/data/instance:
uuid: "6d96519ada1048edaa2437e063b877f0"
format_stamp: "Formatted at 2026-08-12 05:24:19 on dist-test-slave-42z9"
server_key: "c13779cc2a3069a338252a62411ab3ff"
server_key_iv: "77e6b540f68babbb3ef9d408f7148a07"
server_key_version: "encryptionkey@0"
I20260812 05:24:19.107066 32645 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786512017613366-32144-0/minicluster-data/ts-1/wal/instance:
uuid: "6d96519ada1048edaa2437e063b877f0"
format_stamp: "Formatted at 2026-08-12 05:24:19 on dist-test-slave-42z9"
server_key: "c13779cc2a3069a338252a62411ab3ff"
server_key_iv: "77e6b540f68babbb3ef9d408f7148a07"
server_key_version: "encryptionkey@0"
I20260812 05:24:19.108261 32645 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260812 05:24:19.108877 32673 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 05:24:19.109010 32645 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260812 05:24:19.109097 32645 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786512017613366-32144-0/minicluster-data/ts-1/data,/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786512017613366-32144-0/minicluster-data/ts-1/wal
uuid: "6d96519ada1048edaa2437e063b877f0"
format_stamp: "Formatted at 2026-08-12 05:24:19 on dist-test-slave-42z9"
server_key: "c13779cc2a3069a338252a62411ab3ff"
server_key_iv: "77e6b540f68babbb3ef9d408f7148a07"
server_key_version: "encryptionkey@0"
I20260812 05:24:19.109156 32645 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786512017613366-32144-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786512017613366-32144-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786512017613366-32144-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 05:24:19.117105 32645 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 05:24:19.118729 32645 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 05:24:19.118854 32645 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 05:24:19.119055 32645 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 05:24:19.119338 32645 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 05:24:19.119390 32645 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 05:24:19.119418 32645 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 05:24:19.119441 32645 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 05:24:19.124840 32645 rpc_server.cc:307] RPC server started. Bound to: 127.31.100.2:46245
I20260812 05:24:19.124904 421 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.100.2:46245 every 8 connection(s)
I20260812 05:24:19.125170 32645 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786512017613366-32144-0/minicluster-data/ts-1/data/info.pb
I20260812 05:24:19.126181 32144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0waFvT/build/release/bin/kudu as pid 32645
I20260812 05:24:19.126278 32144 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786512017613366-32144-0/minicluster-data/ts-1/wal/instance
I20260812 05:24:19.126449 32144 external_mini_cluster.cc:1428] Setting key eb1d53e6001a4389120f00486b3099d5
Aug 12 05:24:19 dist-test-slave-42z9 krb5kdc[32257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512259, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.100.2@KRBTEST.COM for kudu/127.31.100.62@KRBTEST.COM
I20260812 05:24:19.134557 32390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:24:19.125811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.100.2:58569 (local address 127.31.100.62:42887)
0812 05:24:19.125885 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0812 05:24:19.125888 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 05:24:19.126049 (+ 161us) server_negotiation.cc:407] Connection header received
0812 05:24:19.126576 (+ 527us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:24:19.126578 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:24:19.126590 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:24:19.126653 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:24:19.127838 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:24:19.128334 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:24:19.129715 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:24:19.129911 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:24:19.129969 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:24:19.131961 (+ 1992us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 05:24:19.131963 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 05:24:19.131963 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 05:24:19.131970 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 05:24:19.133263 (+ 1293us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:24:19.133686 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:24:19.133687 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:24:19.133687 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:24:19.133741 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:24:19.133958 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:24:19.133959 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:24:19.133959 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:24:19.134068 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 05:24:19.134124 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0812 05:24:19.134384 (+ 260us) server_negotiation.cc:299] Negotiation successful
0812 05:24:19.134467 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
WARNING: no policy specified for kudu/127.31.100.3@KRBTEST.COM; defaulting to no policy
I20260812 05:24:19.134712 424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:24:19.125776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.62:42887 (local address 127.31.100.2:58569)
0812 05:24:19.125936 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0812 05:24:19.125970 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0812 05:24:19.126482 (+ 512us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:24:19.126757 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:24:19.126761 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:24:19.127554 (+ 793us) client_negotiation.cc:189] Negotiated authn=SASL
0812 05:24:19.127744 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:24:19.127748 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:24:19.128417 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:24:19.128419 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:24:19.129628 (+ 1209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:24:19.129631 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:24:19.129987 (+ 356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:24:19.129988 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:24:19.130096 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:24:19.130568 (+ 472us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 05:24:19.130576 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0812 05:24:19.131872 (+ 1296us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 05:24:19.133341 (+ 1469us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 05:24:19.133348 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 05:24:19.133350 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 05:24:19.133618 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 05:24:19.133796 (+ 178us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 05:24:19.133797 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 05:24:19.133797 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 05:24:19.133903 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 05:24:19.134128 (+ 225us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 05:24:19.134134 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 05:24:19.134264 (+ 130us) client_negotiation.cc:769] Sending connection context
0812 05:24:19.134350 (+ 86us) client_negotiation.cc:240] Negotiation successful
0812 05:24:19.134504 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":85,"threads_started":1}
I20260812 05:24:19.135159 422 heartbeater.cc:347] Connected to a master server at 127.31.100.62:42887
I20260812 05:24:19.135246 422 heartbeater.cc:464] Registering TS with master...
I20260812 05:24:19.135411 422 heartbeater.cc:511] Master 127.31.100.62:42887 requested a full tablet report, sending...
I20260812 05:24:19.135864 32325 ts_manager.cc:194] Registered new tserver with Master: 6d96519ada1048edaa2437e063b877f0 (127.31.100.2:46245)
I20260812 05:24:19.136229 32325 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.100.2@KRBTEST.COM'} at 127.31.100.2:58569
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.100.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.100.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.3.keytab.
Entry for principal kudu/127.31.100.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.3.keytab.
I20260812 05:24:19.149892 32144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0waFvT/build/release/bin/kudu
/tmp/dist-test-task0waFvT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786512017613366-32144-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786512017613366-32144-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786512017613366-32144-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786512017613366-32144-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.100.3:0
--local_ip_for_outbound_sockets=127.31.100.3
--webserver_interface=127.31.100.3
--webserver_port=0
--tserver_master_addrs=127.31.100.62:42887
--keytab_file=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.3.keytab
--principal=kudu/127.31.100.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.100.20:35063
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786512017613366-32144-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786512017613366-32144-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786512017613366-32144-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 05:24:19.233220 431 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 05:24:19.233417 431 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 05:24:19.233455 431 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 05:24:19.234570 431 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 05:24:19.234607 431 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 05:24:19.234632 431 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.100.3
I20260812 05:24:19.236012 431 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.100.20:35063
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786512017613366-32144-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786512017613366-32144-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.3.keytab
--principal=kudu/127.31.100.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.100.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786512017613366-32144-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.100.3
--webserver_port=0
--tserver_master_addrs=127.31.100.62:42887
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.431
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.100.3
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786512017613366-32144-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 04:20:34 UTC on ccycloud.quasar-phqdqp.root.comops.site
build id 4581871
I20260812 05:24:19.236186 431 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 05:24:19.236363 431 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 05:24:19.241026 441 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 05:24:19.241066 444 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 05:24:19.241214 431 server_base.cc:1034] running on GCE node
W20260812 05:24:19.241022 440 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 05:24:19.241508 431 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 05:24:19.241756 431 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 05:24:19.242885 431 hybrid_clock.cc:648] HybridClock initialized: now 1786512259242870 us; error 35 us; skew 500 ppm
Aug 12 05:24:19 dist-test-slave-42z9 krb5kdc[32257](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512259, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.100.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 05:24:19.245261 431 init.cc:377] Logged in from keytab as kudu/127.31.100.3@KRBTEST.COM (short username kudu)
I20260812 05:24:19.245711 431 webserver.cc:460] Webserver started at http://127.31.100.3:45379/ using document root <none> and password file <none>
I20260812 05:24:19.245900 431 fs_manager.cc:359] Metadata directory not provided
I20260812 05:24:19.245940 431 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 05:24:19.246068 431 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 05:24:19.247078 431 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786512017613366-32144-0/minicluster-data/ts-2/data/instance:
uuid: "12b2ec63340b4698807199189532ba6c"
format_stamp: "Formatted at 2026-08-12 05:24:19 on dist-test-slave-42z9"
server_key: "4d1559fd505674417237e5dfe15a17ec"
server_key_iv: "a47c68b89ef801c91ff80b405ee31fc3"
server_key_version: "encryptionkey@0"
I20260812 05:24:19.247392 431 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786512017613366-32144-0/minicluster-data/ts-2/wal/instance:
uuid: "12b2ec63340b4698807199189532ba6c"
format_stamp: "Formatted at 2026-08-12 05:24:19 on dist-test-slave-42z9"
server_key: "4d1559fd505674417237e5dfe15a17ec"
server_key_iv: "a47c68b89ef801c91ff80b405ee31fc3"
server_key_version: "encryptionkey@0"
I20260812 05:24:19.248602 431 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260812 05:24:19.249208 462 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 05:24:19.249387 431 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260812 05:24:19.249500 431 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786512017613366-32144-0/minicluster-data/ts-2/data,/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786512017613366-32144-0/minicluster-data/ts-2/wal
uuid: "12b2ec63340b4698807199189532ba6c"
format_stamp: "Formatted at 2026-08-12 05:24:19 on dist-test-slave-42z9"
server_key: "4d1559fd505674417237e5dfe15a17ec"
server_key_iv: "a47c68b89ef801c91ff80b405ee31fc3"
server_key_version: "encryptionkey@0"
I20260812 05:24:19.249563 431 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786512017613366-32144-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786512017613366-32144-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786512017613366-32144-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 05:24:19.264660 431 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 05:24:19.266762 431 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 05:24:19.266875 431 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 05:24:19.267062 431 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 05:24:19.267340 431 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 05:24:19.267371 431 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 05:24:19.267393 431 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 05:24:19.267409 431 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 05:24:19.272713 431 rpc_server.cc:307] RPC server started. Bound to: 127.31.100.3:42909
I20260812 05:24:19.272784 670 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.100.3:42909 every 8 connection(s)
I20260812 05:24:19.273064 431 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786512017613366-32144-0/minicluster-data/ts-2/data/info.pb
I20260812 05:24:19.275149 32144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0waFvT/build/release/bin/kudu as pid 431
I20260812 05:24:19.275239 32144 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786512017613366-32144-0/minicluster-data/ts-2/wal/instance
I20260812 05:24:19.275405 32144 external_mini_cluster.cc:1428] Setting key 673f73d77a7c5e6b581dcff5cb703dc6
Aug 12 05:24:19 dist-test-slave-42z9 krb5kdc[32257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512259, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.100.3@KRBTEST.COM for kudu/127.31.100.62@KRBTEST.COM
I20260812 05:24:19.281709 32390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:24:19.273732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.100.3:36893 (local address 127.31.100.62:42887)
0812 05:24:19.273790 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0812 05:24:19.273792 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 05:24:19.273966 (+ 174us) server_negotiation.cc:407] Connection header received
0812 05:24:19.274419 (+ 453us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:24:19.274420 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:24:19.274430 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:24:19.274486 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:24:19.275461 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:24:19.276167 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:24:19.277155 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:24:19.277345 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:24:19.277394 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:24:19.279408 (+ 2014us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 05:24:19.279410 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 05:24:19.279410 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 05:24:19.279417 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 05:24:19.280637 (+ 1220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:24:19.280969 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:24:19.280970 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:24:19.280970 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:24:19.281016 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:24:19.281188 (+ 172us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:24:19.281188 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:24:19.281189 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:24:19.281285 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 05:24:19.281329 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0812 05:24:19.281575 (+ 246us) server_negotiation.cc:299] Negotiation successful
0812 05:24:19.281640 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260812 05:24:19.281891 676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:24:19.273691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.62:42887 (local address 127.31.100.3:36893)
0812 05:24:19.273844 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0812 05:24:19.273886 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0812 05:24:19.274343 (+ 457us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:24:19.274563 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:24:19.274566 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:24:19.275138 (+ 572us) client_negotiation.cc:189] Negotiated authn=SASL
0812 05:24:19.275365 (+ 227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:24:19.275371 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:24:19.276257 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:24:19.276258 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:24:19.277081 (+ 823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:24:19.277084 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:24:19.277404 (+ 320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:24:19.277405 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:24:19.277554 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:24:19.278003 (+ 449us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 05:24:19.278011 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0812 05:24:19.279305 (+ 1294us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 05:24:19.280707 (+ 1402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 05:24:19.280712 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 05:24:19.280714 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 05:24:19.280914 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 05:24:19.281058 (+ 144us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 05:24:19.281058 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 05:24:19.281059 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 05:24:19.281147 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 05:24:19.281326 (+ 179us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 05:24:19.281329 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 05:24:19.281433 (+ 104us) client_negotiation.cc:769] Sending connection context
0812 05:24:19.281562 (+ 129us) client_negotiation.cc:240] Negotiation successful
0812 05:24:19.281683 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":69,"threads_started":1}
I20260812 05:24:19.282330 672 heartbeater.cc:347] Connected to a master server at 127.31.100.62:42887
I20260812 05:24:19.282416 672 heartbeater.cc:464] Registering TS with master...
I20260812 05:24:19.282593 672 heartbeater.cc:511] Master 127.31.100.62:42887 requested a full tablet report, sending...
I20260812 05:24:19.282953 32325 ts_manager.cc:194] Registered new tserver with Master: 12b2ec63340b4698807199189532ba6c (127.31.100.3:42909)
I20260812 05:24:19.283310 32325 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.100.3@KRBTEST.COM'} at 127.31.100.3:36893
I20260812 05:24:19.286895 32144 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 05:24:19.293390 32390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:24:19.288208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40806 (local address 127.31.100.62:42887)
0812 05:24:19.288269 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0812 05:24:19.288270 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0812 05:24:19.288353 (+ 83us) server_negotiation.cc:407] Connection header received
0812 05:24:19.288401 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:24:19.288402 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:24:19.288412 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:24:19.288456 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:24:19.288995 (+ 539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:24:19.289517 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:24:19.290255 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:24:19.290432 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:24:19.290479 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:24:19.291515 (+ 1036us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 05:24:19.291516 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 05:24:19.291516 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 05:24:19.291520 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0812 05:24:19.292550 (+ 1030us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:24:19.292827 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:24:19.292828 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:24:19.292828 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:24:19.292871 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:24:19.292994 (+ 123us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:24:19.292994 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:24:19.292994 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:24:19.293124 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 05:24:19.293195 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0812 05:24:19.293268 (+ 73us) server_negotiation.cc:299] Negotiation successful
0812 05:24:19.293327 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260812 05:24:19.294855 32325 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40806:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 05:24:19.296445 32325 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 05:24:19.302755 688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:24:19.298479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.3:42909 (local address 127.0.0.1:39468)
0812 05:24:19.298854 (+ 375us) negotiation.cc:107] Waiting for socket to connect
0812 05:24:19.298863 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0812 05:24:19.298901 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:24:19.299075 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:24:19.299078 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:24:19.299080 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 05:24:19.299151 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:24:19.299153 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:24:19.300317 (+ 1164us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:24:19.300318 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:24:19.301585 (+ 1267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:24:19.301588 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:24:19.302487 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:24:19.302487 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:24:19.302549 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:24:19.302562 (+ 13us) client_negotiation.cc:769] Sending connection context
0812 05:24:19.302626 (+ 64us) client_negotiation.cc:240] Negotiation successful
0812 05:24:19.302679 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":358,"thread_start_us":67,"threads_started":1}
I20260812 05:24:19.302834 689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:24:19.298584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59316 (local address 127.31.100.2:46245)
0812 05:24:19.299125 (+ 541us) server_negotiation.cc:206] Beginning negotiation
0812 05:24:19.299129 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 05:24:19.299137 (+ 8us) server_negotiation.cc:407] Connection header received
0812 05:24:19.299172 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:24:19.299174 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:24:19.299187 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:24:19.299264 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 05:24:19.299352 (+ 88us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:24:19.300533 (+ 1181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:24:19.301826 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:24:19.302521 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:24:19.302583 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:24:19.302636 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0812 05:24:19.302720 (+ 84us) server_negotiation.cc:299] Negotiation successful
0812 05:24:19.302759 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":500,"thread_start_us":77,"threads_started":1}
I20260812 05:24:19.302830 685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:24:19.298479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.2:46245 (local address 127.0.0.1:59316)
0812 05:24:19.298936 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0812 05:24:19.298947 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0812 05:24:19.298998 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:24:19.299261 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:24:19.299262 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:24:19.299264 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 05:24:19.299297 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:24:19.299298 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:24:19.300634 (+ 1336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:24:19.300635 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:24:19.301757 (+ 1122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:24:19.301760 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:24:19.302587 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:24:19.302588 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:24:19.302658 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:24:19.302662 (+ 4us) client_negotiation.cc:769] Sending connection context
0812 05:24:19.302708 (+ 46us) client_negotiation.cc:240] Negotiation successful
0812 05:24:19.302739 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":415,"mutex_wait_us":31,"thread_start_us":77,"threads_started":1}
I20260812 05:24:19.303391 691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:24:19.298726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39468 (local address 127.31.100.3:42909)
0812 05:24:19.298924 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0812 05:24:19.298928 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 05:24:19.298937 (+ 9us) server_negotiation.cc:407] Connection header received
0812 05:24:19.298974 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:24:19.298976 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:24:19.298990 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:24:19.299058 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 05:24:19.299222 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:24:19.300222 (+ 1000us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:24:19.301671 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:24:19.302408 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:24:19.302478 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:24:19.302539 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0812 05:24:19.303273 (+ 734us) server_negotiation.cc:299] Negotiation successful
0812 05:24:19.303318 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":169,"thread_start_us":92,"threads_started":1}
I20260812 05:24:19.303568 687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:24:19.298476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.1:33067 (local address 127.0.0.1:58322)
0812 05:24:19.298651 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0812 05:24:19.298690 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0812 05:24:19.298754 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:24:19.299169 (+ 415us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:24:19.299170 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:24:19.299172 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 05:24:19.299245 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:24:19.299247 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:24:19.300650 (+ 1403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:24:19.300651 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:24:19.301596 (+ 945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:24:19.301598 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:24:19.303338 (+ 1740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:24:19.303339 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:24:19.303401 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:24:19.303404 (+ 3us) client_negotiation.cc:769] Sending connection context
0812 05:24:19.303449 (+ 45us) client_negotiation.cc:240] Negotiation successful
0812 05:24:19.303490 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"mutex_wait_us":21,"thread_start_us":56,"threads_started":1}
I20260812 05:24:19.303928 690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:24:19.298729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58322 (local address 127.31.100.1:33067)
0812 05:24:19.298917 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0812 05:24:19.298922 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 05:24:19.298933 (+ 11us) server_negotiation.cc:407] Connection header received
0812 05:24:19.298993 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:24:19.298995 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:24:19.299010 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:24:19.299087 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 05:24:19.299327 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:24:19.300524 (+ 1197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:24:19.302588 (+ 2064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:24:19.303270 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:24:19.303329 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:24:19.303392 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0812 05:24:19.303809 (+ 417us) server_negotiation.cc:299] Negotiation successful
0812 05:24:19.303851 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":86,"threads_started":1}
I20260812 05:24:19.303812 303 tablet_service.cc:1467] Processing CreateTablet for tablet c343a418e285412aa996f678cdde0c02 (DEFAULT_TABLE table=test-table [id=44f1d20b7f44442591b2a74d498c5075]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 05:24:19.304211 303 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c343a418e285412aa996f678cdde0c02. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 05:24:19.304551 32534 tablet_service.cc:1467] Processing CreateTablet for tablet c343a418e285412aa996f678cdde0c02 (DEFAULT_TABLE table=test-table [id=44f1d20b7f44442591b2a74d498c5075]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 05:24:19.304838 32534 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c343a418e285412aa996f678cdde0c02. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 05:24:19.307224 694 tablet_bootstrap.cc:492] T c343a418e285412aa996f678cdde0c02 P 6d96519ada1048edaa2437e063b877f0: Bootstrap starting.
I20260812 05:24:19.307785 555 tablet_service.cc:1467] Processing CreateTablet for tablet c343a418e285412aa996f678cdde0c02 (DEFAULT_TABLE table=test-table [id=44f1d20b7f44442591b2a74d498c5075]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 05:24:19.308130 555 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c343a418e285412aa996f678cdde0c02. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 05:24:19.308292 694 tablet_bootstrap.cc:654] T c343a418e285412aa996f678cdde0c02 P 6d96519ada1048edaa2437e063b877f0: Neither blocks nor log segments found. Creating new log.
I20260812 05:24:19.308836 694 log.cc:824] T c343a418e285412aa996f678cdde0c02 P 6d96519ada1048edaa2437e063b877f0: Log is configured to *not* fsync() on all Append() calls
I20260812 05:24:19.309217 695 tablet_bootstrap.cc:492] T c343a418e285412aa996f678cdde0c02 P 4b4c5f4c2c1d444b8550f241e84a182e: Bootstrap starting.
I20260812 05:24:19.309713 694 tablet_bootstrap.cc:492] T c343a418e285412aa996f678cdde0c02 P 6d96519ada1048edaa2437e063b877f0: No bootstrap required, opened a new log
I20260812 05:24:19.309790 694 ts_tablet_manager.cc:1397] T c343a418e285412aa996f678cdde0c02 P 6d96519ada1048edaa2437e063b877f0: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260812 05:24:19.311349 698 tablet_bootstrap.cc:492] T c343a418e285412aa996f678cdde0c02 P 12b2ec63340b4698807199189532ba6c: Bootstrap starting.
I20260812 05:24:19.311724 694 raft_consensus.cc:348] T c343a418e285412aa996f678cdde0c02 P 6d96519ada1048edaa2437e063b877f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d96519ada1048edaa2437e063b877f0" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 46245 } } peers { permanent_uuid: "4b4c5f4c2c1d444b8550f241e84a182e" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 33067 } } peers { permanent_uuid: "12b2ec63340b4698807199189532ba6c" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 42909 } }
I20260812 05:24:19.311899 694 raft_consensus.cc:374] T c343a418e285412aa996f678cdde0c02 P 6d96519ada1048edaa2437e063b877f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 05:24:19.311924 694 raft_consensus.cc:729] T c343a418e285412aa996f678cdde0c02 P 6d96519ada1048edaa2437e063b877f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d96519ada1048edaa2437e063b877f0, State: Initialized, Role: FOLLOWER
I20260812 05:24:19.311951 698 tablet_bootstrap.cc:654] T c343a418e285412aa996f678cdde0c02 P 12b2ec63340b4698807199189532ba6c: Neither blocks nor log segments found. Creating new log.
I20260812 05:24:19.312047 694 consensus_queue.cc:260] T c343a418e285412aa996f678cdde0c02 P 6d96519ada1048edaa2437e063b877f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d96519ada1048edaa2437e063b877f0" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 46245 } } peers { permanent_uuid: "4b4c5f4c2c1d444b8550f241e84a182e" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 33067 } } peers { permanent_uuid: "12b2ec63340b4698807199189532ba6c" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 42909 } }
I20260812 05:24:19.312320 694 ts_tablet_manager.cc:1428] T c343a418e285412aa996f678cdde0c02 P 6d96519ada1048edaa2437e063b877f0: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260812 05:24:19.312377 422 heartbeater.cc:503] Master 127.31.100.62:42887 was elected leader, sending a full tablet report...
I20260812 05:24:19.312463 698 log.cc:824] T c343a418e285412aa996f678cdde0c02 P 12b2ec63340b4698807199189532ba6c: Log is configured to *not* fsync() on all Append() calls
I20260812 05:24:19.312933 695 tablet_bootstrap.cc:654] T c343a418e285412aa996f678cdde0c02 P 4b4c5f4c2c1d444b8550f241e84a182e: Neither blocks nor log segments found. Creating new log.
I20260812 05:24:19.313318 698 tablet_bootstrap.cc:492] T c343a418e285412aa996f678cdde0c02 P 12b2ec63340b4698807199189532ba6c: No bootstrap required, opened a new log
I20260812 05:24:19.313391 698 ts_tablet_manager.cc:1397] T c343a418e285412aa996f678cdde0c02 P 12b2ec63340b4698807199189532ba6c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260812 05:24:19.313508 695 log.cc:824] T c343a418e285412aa996f678cdde0c02 P 4b4c5f4c2c1d444b8550f241e84a182e: Log is configured to *not* fsync() on all Append() calls
I20260812 05:24:19.314126 695 tablet_bootstrap.cc:492] T c343a418e285412aa996f678cdde0c02 P 4b4c5f4c2c1d444b8550f241e84a182e: No bootstrap required, opened a new log
I20260812 05:24:19.314184 695 ts_tablet_manager.cc:1397] T c343a418e285412aa996f678cdde0c02 P 4b4c5f4c2c1d444b8550f241e84a182e: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.001s
I20260812 05:24:19.314920 698 raft_consensus.cc:348] T c343a418e285412aa996f678cdde0c02 P 12b2ec63340b4698807199189532ba6c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d96519ada1048edaa2437e063b877f0" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 46245 } } peers { permanent_uuid: "4b4c5f4c2c1d444b8550f241e84a182e" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 33067 } } peers { permanent_uuid: "12b2ec63340b4698807199189532ba6c" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 42909 } }
I20260812 05:24:19.315053 698 raft_consensus.cc:374] T c343a418e285412aa996f678cdde0c02 P 12b2ec63340b4698807199189532ba6c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 05:24:19.315076 698 raft_consensus.cc:729] T c343a418e285412aa996f678cdde0c02 P 12b2ec63340b4698807199189532ba6c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12b2ec63340b4698807199189532ba6c, State: Initialized, Role: FOLLOWER
I20260812 05:24:19.315164 698 consensus_queue.cc:260] T c343a418e285412aa996f678cdde0c02 P 12b2ec63340b4698807199189532ba6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d96519ada1048edaa2437e063b877f0" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 46245 } } peers { permanent_uuid: "4b4c5f4c2c1d444b8550f241e84a182e" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 33067 } } peers { permanent_uuid: "12b2ec63340b4698807199189532ba6c" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 42909 } }
I20260812 05:24:19.315404 698 ts_tablet_manager.cc:1428] T c343a418e285412aa996f678cdde0c02 P 12b2ec63340b4698807199189532ba6c: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260812 05:24:19.315429 672 heartbeater.cc:503] Master 127.31.100.62:42887 was elected leader, sending a full tablet report...
I20260812 05:24:19.315652 695 raft_consensus.cc:348] T c343a418e285412aa996f678cdde0c02 P 4b4c5f4c2c1d444b8550f241e84a182e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d96519ada1048edaa2437e063b877f0" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 46245 } } peers { permanent_uuid: "4b4c5f4c2c1d444b8550f241e84a182e" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 33067 } } peers { permanent_uuid: "12b2ec63340b4698807199189532ba6c" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 42909 } }
I20260812 05:24:19.315830 695 raft_consensus.cc:374] T c343a418e285412aa996f678cdde0c02 P 4b4c5f4c2c1d444b8550f241e84a182e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 05:24:19.315873 695 raft_consensus.cc:729] T c343a418e285412aa996f678cdde0c02 P 4b4c5f4c2c1d444b8550f241e84a182e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b4c5f4c2c1d444b8550f241e84a182e, State: Initialized, Role: FOLLOWER
I20260812 05:24:19.315976 695 consensus_queue.cc:260] T c343a418e285412aa996f678cdde0c02 P 4b4c5f4c2c1d444b8550f241e84a182e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d96519ada1048edaa2437e063b877f0" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 46245 } } peers { permanent_uuid: "4b4c5f4c2c1d444b8550f241e84a182e" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 33067 } } peers { permanent_uuid: "12b2ec63340b4698807199189532ba6c" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 42909 } }
I20260812 05:24:19.316262 695 ts_tablet_manager.cc:1428] T c343a418e285412aa996f678cdde0c02 P 4b4c5f4c2c1d444b8550f241e84a182e: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260812 05:24:19.316275 32637 heartbeater.cc:503] Master 127.31.100.62:42887 was elected leader, sending a full tablet report...
I20260812 05:24:19.348330 699 raft_consensus.cc:482] T c343a418e285412aa996f678cdde0c02 P 6d96519ada1048edaa2437e063b877f0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 05:24:19.348465 699 raft_consensus.cc:504] T c343a418e285412aa996f678cdde0c02 P 6d96519ada1048edaa2437e063b877f0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d96519ada1048edaa2437e063b877f0" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 46245 } } peers { permanent_uuid: "4b4c5f4c2c1d444b8550f241e84a182e" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 33067 } } peers { permanent_uuid: "12b2ec63340b4698807199189532ba6c" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 42909 } }
I20260812 05:24:19.348843 699 leader_election.cc:288] T c343a418e285412aa996f678cdde0c02 P 6d96519ada1048edaa2437e063b877f0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4b4c5f4c2c1d444b8550f241e84a182e (127.31.100.1:33067), 12b2ec63340b4698807199189532ba6c (127.31.100.3:42909)
I20260812 05:24:19.352609 690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:24:19.349159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.100.2:42239 (local address 127.31.100.1:33067)
0812 05:24:19.349209 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0812 05:24:19.349211 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 05:24:19.349221 (+ 10us) server_negotiation.cc:407] Connection header received
0812 05:24:19.349247 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:24:19.349248 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:24:19.349257 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:24:19.349323 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 05:24:19.349512 (+ 189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:24:19.350269 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:24:19.351659 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:24:19.352272 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:24:19.352342 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:24:19.352350 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0812 05:24:19.352505 (+ 155us) server_negotiation.cc:299] Negotiation successful
0812 05:24:19.352543 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260812 05:24:19.352627 691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:24:19.349145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.100.2:59145 (local address 127.31.100.3:42909)
0812 05:24:19.349215 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0812 05:24:19.349217 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 05:24:19.349278 (+ 61us) server_negotiation.cc:407] Connection header received
0812 05:24:19.349305 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:24:19.349306 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:24:19.349314 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:24:19.349367 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 05:24:19.349502 (+ 135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:24:19.350263 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:24:19.351658 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:24:19.352269 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:24:19.352341 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:24:19.352350 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0812 05:24:19.352505 (+ 155us) server_negotiation.cc:299] Negotiation successful
0812 05:24:19.352574 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260812 05:24:19.352639 424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:24:19.349024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.1:33067 (local address 127.31.100.2:42239)
0812 05:24:19.349086 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0812 05:24:19.349097 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0812 05:24:19.349139 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:24:19.349339 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:24:19.349340 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:24:19.349341 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 05:24:19.349422 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:24:19.349424 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:24:19.350339 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:24:19.350339 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:24:19.351579 (+ 1240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:24:19.351583 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:24:19.352344 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:24:19.352345 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:24:19.352433 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:24:19.352435 (+ 2us) client_negotiation.cc:769] Sending connection context
0812 05:24:19.352494 (+ 59us) client_negotiation.cc:240] Negotiation successful
0812 05:24:19.352535 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42,"mutex_wait_us":10}
I20260812 05:24:19.352644 714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:24:19.349054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.3:42909 (local address 127.31.100.2:59145)
0812 05:24:19.349221 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0812 05:24:19.349228 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0812 05:24:19.349271 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:24:19.349372 (+ 101us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:24:19.349373 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:24:19.349374 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 05:24:19.349423 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:24:19.349424 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:24:19.350338 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:24:19.350339 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:24:19.351586 (+ 1247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:24:19.351588 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:24:19.352345 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:24:19.352346 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:24:19.352430 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:24:19.352435 (+ 5us) client_negotiation.cc:769] Sending connection context
0812 05:24:19.352494 (+ 59us) client_negotiation.cc:240] Negotiation successful
0812 05:24:19.352535 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154,"mutex_wait_us":78,"thread_start_us":113,"threads_started":1}
I20260812 05:24:19.353006 588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c343a418e285412aa996f678cdde0c02" candidate_uuid: "6d96519ada1048edaa2437e063b877f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12b2ec63340b4698807199189532ba6c" is_pre_election: true
I20260812 05:24:19.353025 32568 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c343a418e285412aa996f678cdde0c02" candidate_uuid: "6d96519ada1048edaa2437e063b877f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4b4c5f4c2c1d444b8550f241e84a182e" is_pre_election: true
I20260812 05:24:19.353152 588 raft_consensus.cc:2454] T c343a418e285412aa996f678cdde0c02 P 12b2ec63340b4698807199189532ba6c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d96519ada1048edaa2437e063b877f0 in term 0.
I20260812 05:24:19.353156 32568 raft_consensus.cc:2454] T c343a418e285412aa996f678cdde0c02 P 4b4c5f4c2c1d444b8550f241e84a182e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d96519ada1048edaa2437e063b877f0 in term 0.
I20260812 05:24:19.353370 32679 leader_election.cc:302] T c343a418e285412aa996f678cdde0c02 P 6d96519ada1048edaa2437e063b877f0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12b2ec63340b4698807199189532ba6c, 6d96519ada1048edaa2437e063b877f0; no voters:
I20260812 05:24:19.353574 699 raft_consensus.cc:2781] T c343a418e285412aa996f678cdde0c02 P 6d96519ada1048edaa2437e063b877f0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 05:24:19.353614 699 raft_consensus.cc:482] T c343a418e285412aa996f678cdde0c02 P 6d96519ada1048edaa2437e063b877f0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 05:24:19.353628 699 raft_consensus.cc:3037] T c343a418e285412aa996f678cdde0c02 P 6d96519ada1048edaa2437e063b877f0 [term 0 FOLLOWER]: Advancing to term 1
I20260812 05:24:19.354298 699 raft_consensus.cc:504] T c343a418e285412aa996f678cdde0c02 P 6d96519ada1048edaa2437e063b877f0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d96519ada1048edaa2437e063b877f0" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 46245 } } peers { permanent_uuid: "4b4c5f4c2c1d444b8550f241e84a182e" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 33067 } } peers { permanent_uuid: "12b2ec63340b4698807199189532ba6c" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 42909 } }
I20260812 05:24:19.354416 699 leader_election.cc:288] T c343a418e285412aa996f678cdde0c02 P 6d96519ada1048edaa2437e063b877f0 [CANDIDATE]: Term 1 election: Requested vote from peers 4b4c5f4c2c1d444b8550f241e84a182e (127.31.100.1:33067), 12b2ec63340b4698807199189532ba6c (127.31.100.3:42909)
I20260812 05:24:19.354580 32568 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c343a418e285412aa996f678cdde0c02" candidate_uuid: "6d96519ada1048edaa2437e063b877f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4b4c5f4c2c1d444b8550f241e84a182e"
I20260812 05:24:19.354580 588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c343a418e285412aa996f678cdde0c02" candidate_uuid: "6d96519ada1048edaa2437e063b877f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12b2ec63340b4698807199189532ba6c"
I20260812 05:24:19.354665 588 raft_consensus.cc:3037] T c343a418e285412aa996f678cdde0c02 P 12b2ec63340b4698807199189532ba6c [term 0 FOLLOWER]: Advancing to term 1
I20260812 05:24:19.354665 32568 raft_consensus.cc:3037] T c343a418e285412aa996f678cdde0c02 P 4b4c5f4c2c1d444b8550f241e84a182e [term 0 FOLLOWER]: Advancing to term 1
I20260812 05:24:19.355564 588 raft_consensus.cc:2454] T c343a418e285412aa996f678cdde0c02 P 12b2ec63340b4698807199189532ba6c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d96519ada1048edaa2437e063b877f0 in term 1.
I20260812 05:24:19.355564 32568 raft_consensus.cc:2454] T c343a418e285412aa996f678cdde0c02 P 4b4c5f4c2c1d444b8550f241e84a182e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d96519ada1048edaa2437e063b877f0 in term 1.
I20260812 05:24:19.355721 32679 leader_election.cc:302] T c343a418e285412aa996f678cdde0c02 P 6d96519ada1048edaa2437e063b877f0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12b2ec63340b4698807199189532ba6c, 6d96519ada1048edaa2437e063b877f0; no voters:
I20260812 05:24:19.355799 699 raft_consensus.cc:2781] T c343a418e285412aa996f678cdde0c02 P 6d96519ada1048edaa2437e063b877f0 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 05:24:19.355932 699 raft_consensus.cc:686] T c343a418e285412aa996f678cdde0c02 P 6d96519ada1048edaa2437e063b877f0 [term 1 LEADER]: Becoming Leader. State: Replica: 6d96519ada1048edaa2437e063b877f0, State: Running, Role: LEADER
I20260812 05:24:19.356024 699 consensus_queue.cc:237] T c343a418e285412aa996f678cdde0c02 P 6d96519ada1048edaa2437e063b877f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d96519ada1048edaa2437e063b877f0" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 46245 } } peers { permanent_uuid: "4b4c5f4c2c1d444b8550f241e84a182e" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 33067 } } peers { permanent_uuid: "12b2ec63340b4698807199189532ba6c" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 42909 } }
I20260812 05:24:19.356805 32325 catalog_manager.cc:5495] T c343a418e285412aa996f678cdde0c02 P 6d96519ada1048edaa2437e063b877f0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6d96519ada1048edaa2437e063b877f0 (127.31.100.2). New cstate: current_term: 1 leader_uuid: "6d96519ada1048edaa2437e063b877f0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d96519ada1048edaa2437e063b877f0" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 46245 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4b4c5f4c2c1d444b8550f241e84a182e" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 33067 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "12b2ec63340b4698807199189532ba6c" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 42909 } health_report { overall_health: UNKNOWN } } }
I20260812 05:24:19.374138 689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:24:19.372015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59320 (local address 127.31.100.2:46245)
0812 05:24:19.372057 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0812 05:24:19.372059 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 05:24:19.372067 (+ 8us) server_negotiation.cc:407] Connection header received
0812 05:24:19.372141 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:24:19.372142 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:24:19.372159 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:24:19.372215 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 05:24:19.372320 (+ 105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:24:19.372780 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:24:19.373495 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:24:19.373694 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:24:19.373747 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:24:19.373854 (+ 107us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 05:24:19.373948 (+ 94us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 05:24:19.374002 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0812 05:24:19.374021 (+ 19us) server_negotiation.cc:299] Negotiation successful
0812 05:24:19.374040 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
W20260812 05:24:19.375669 423 tablet.cc:2369] T c343a418e285412aa996f678cdde0c02 P 6d96519ada1048edaa2437e063b877f0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 05:24:19.375901 32568 raft_consensus.cc:1261] T c343a418e285412aa996f678cdde0c02 P 4b4c5f4c2c1d444b8550f241e84a182e [term 1 FOLLOWER]: Refusing update from remote peer 6d96519ada1048edaa2437e063b877f0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 05:24:19.375902 588 raft_consensus.cc:1261] T c343a418e285412aa996f678cdde0c02 P 12b2ec63340b4698807199189532ba6c [term 1 FOLLOWER]: Refusing update from remote peer 6d96519ada1048edaa2437e063b877f0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 05:24:19.376245 716 consensus_queue.cc:1035] T c343a418e285412aa996f678cdde0c02 P 6d96519ada1048edaa2437e063b877f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b4c5f4c2c1d444b8550f241e84a182e" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 33067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 05:24:19.376364 699 consensus_queue.cc:1035] T c343a418e285412aa996f678cdde0c02 P 6d96519ada1048edaa2437e063b877f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "12b2ec63340b4698807199189532ba6c" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 42909 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 05:24:19.379185 725 mvcc.cc:205] Tried to move back new op lower bound from 7317554214401183744 to 7317554214322909184. Current Snapshot: MvccSnapshot[applied={T|T < 7317554214401183744}]
I20260812 05:24:19.379305 724 mvcc.cc:205] Tried to move back new op lower bound from 7317554214401183744 to 7317554214322909184. Current Snapshot: MvccSnapshot[applied={T|T < 7317554214401183744}]
I20260812 05:24:19.381134 32324 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40806:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 05:24:19.381242 32324 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40806:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 05:24:19.382203 32324 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 77445fd1d2d04cbfb61323303f539507: Sending DeleteTablet for 3 replicas of tablet c343a418e285412aa996f678cdde0c02
I20260812 05:24:19.382519 303 tablet_service.cc:1514] Processing DeleteTablet for tablet c343a418e285412aa996f678cdde0c02 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 05:24:19 UTC) from {username='kudu', principal='kudu/127.31.100.62@KRBTEST.COM'} at 127.0.0.1:59316
I20260812 05:24:19.382670 555 tablet_service.cc:1514] Processing DeleteTablet for tablet c343a418e285412aa996f678cdde0c02 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 05:24:19 UTC) from {username='kudu', principal='kudu/127.31.100.62@KRBTEST.COM'} at 127.0.0.1:39468
I20260812 05:24:19.382738 32534 tablet_service.cc:1514] Processing DeleteTablet for tablet c343a418e285412aa996f678cdde0c02 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 05:24:19 UTC) from {username='kudu', principal='kudu/127.31.100.62@KRBTEST.COM'} at 127.0.0.1:58322
I20260812 05:24:19.382946 32144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0waFvT/build/release/bin/kudu with pid 32432
I20260812 05:24:19.382951 745 tablet_replica.cc:330] T c343a418e285412aa996f678cdde0c02 P 4b4c5f4c2c1d444b8550f241e84a182e: stopping tablet replica
I20260812 05:24:19.383068 745 raft_consensus.cc:2229] T c343a418e285412aa996f678cdde0c02 P 4b4c5f4c2c1d444b8550f241e84a182e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 05:24:19.383172 745 raft_consensus.cc:2258] T c343a418e285412aa996f678cdde0c02 P 4b4c5f4c2c1d444b8550f241e84a182e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 05:24:19.391647 739 tablet_replica.cc:330] T c343a418e285412aa996f678cdde0c02 P 6d96519ada1048edaa2437e063b877f0: stopping tablet replica
I20260812 05:24:19.391796 739 raft_consensus.cc:2229] T c343a418e285412aa996f678cdde0c02 P 6d96519ada1048edaa2437e063b877f0 [term 1 LEADER]: Raft consensus shutting down.
I20260812 05:24:19.391950 742 tablet_replica.cc:330] T c343a418e285412aa996f678cdde0c02 P 12b2ec63340b4698807199189532ba6c: stopping tablet replica
I20260812 05:24:19.391968 739 raft_consensus.cc:2258] T c343a418e285412aa996f678cdde0c02 P 6d96519ada1048edaa2437e063b877f0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 05:24:19.392078 742 raft_consensus.cc:2229] T c343a418e285412aa996f678cdde0c02 P 12b2ec63340b4698807199189532ba6c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 05:24:19.392189 742 raft_consensus.cc:2258] T c343a418e285412aa996f678cdde0c02 P 12b2ec63340b4698807199189532ba6c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 05:24:19.392616 739 ts_tablet_manager.cc:1905] T c343a418e285412aa996f678cdde0c02 P 6d96519ada1048edaa2437e063b877f0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 05:24:19.392652 742 ts_tablet_manager.cc:1905] T c343a418e285412aa996f678cdde0c02 P 12b2ec63340b4698807199189532ba6c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 05:24:19.394301 742 ts_tablet_manager.cc:1918] T c343a418e285412aa996f678cdde0c02 P 12b2ec63340b4698807199189532ba6c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 05:24:19.394383 742 log.cc:1196] T c343a418e285412aa996f678cdde0c02 P 12b2ec63340b4698807199189532ba6c: Deleting WAL directory at /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786512017613366-32144-0/minicluster-data/ts-2/wal/wals/c343a418e285412aa996f678cdde0c02
I20260812 05:24:19.394472 739 ts_tablet_manager.cc:1918] T c343a418e285412aa996f678cdde0c02 P 6d96519ada1048edaa2437e063b877f0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 05:24:19.394611 739 log.cc:1196] T c343a418e285412aa996f678cdde0c02 P 6d96519ada1048edaa2437e063b877f0: Deleting WAL directory at /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786512017613366-32144-0/minicluster-data/ts-1/wal/wals/c343a418e285412aa996f678cdde0c02
I20260812 05:24:19.394658 742 ts_tablet_manager.cc:1939] T c343a418e285412aa996f678cdde0c02 P 12b2ec63340b4698807199189532ba6c: Deleting consensus metadata
I20260812 05:24:19.395041 739 ts_tablet_manager.cc:1939] T c343a418e285412aa996f678cdde0c02 P 6d96519ada1048edaa2437e063b877f0: Deleting consensus metadata
I20260812 05:24:19.395104 32301 catalog_manager.cc:4841] TS 12b2ec63340b4698807199189532ba6c (127.31.100.3:42909): tablet c343a418e285412aa996f678cdde0c02 (table test-table [id=44f1d20b7f44442591b2a74d498c5075]) successfully deleted
I20260812 05:24:19.395591 32303 catalog_manager.cc:4841] TS 6d96519ada1048edaa2437e063b877f0 (127.31.100.2:46245): tablet c343a418e285412aa996f678cdde0c02 (table test-table [id=44f1d20b7f44442591b2a74d498c5075]) successfully deleted
W20260812 05:24:19.396921 32299 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.31.100.1:33067 (error 108)
I20260812 05:24:19.397183 32144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0waFvT/build/release/bin/kudu with pid 32645
I20260812 05:24:19.404801 685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:24:19.397236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.1:33067 (local address 127.0.0.1:58338)
0812 05:24:19.404705 (+ 7469us) negotiation.cc:107] Waiting for socket to connect
0812 05:24:19.404767 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.100.1:33067: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":7443}
W20260812 05:24:19.404958 32299 catalog_manager.cc:4584] TS 4b4c5f4c2c1d444b8550f241e84a182e (127.31.100.1:33067): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c343a418e285412aa996f678cdde0c02: Network error: Client connection negotiation failed: client connection to 127.31.100.1:33067: connect: Connection refused (error 111)
I20260812 05:24:19.409081 32144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0waFvT/build/release/bin/kudu with pid 431
I20260812 05:24:19.424561 32144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0waFvT/build/release/bin/kudu with pid 32275
2026-08-12T05:24:19Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2007 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-08-12T05:24:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T05:24:19Z Disabled control of system clock
I20260812 05:24:19.456952 32144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0waFvT/build/release/bin/kudu
/tmp/dist-test-task0waFvT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786512017613366-32144-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786512017613366-32144-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786512017613366-32144-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786512017613366-32144-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.100.62:34425
--webserver_interface=127.31.100.62
--webserver_port=0
--builtin_ntp_servers=127.31.100.20:39481
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.100.62:34425
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260812 05:24:19.539273 754 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 05:24:19.539507 754 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 05:24:19.539526 754 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 05:24:19.540679 754 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 05:24:19.540719 754 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 05:24:19.540729 754 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 05:24:19.540741 754 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 05:24:19.540766 754 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 05:24:19.542142 754 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.100.20:39481
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786512017613366-32144-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786512017613366-32144-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.100.62:34425
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.100.62:34425
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786512017613366-32144-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.100.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.754
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786512017613366-32144-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 04:20:34 UTC on ccycloud.quasar-phqdqp.root.comops.site
build id 4581871
I20260812 05:24:19.542372 754 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 05:24:19.542644 754 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 05:24:19.547281 759 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 05:24:19.547287 763 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 05:24:19.547351 761 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 05:24:19.547554 754 server_base.cc:1034] running on GCE node
I20260812 05:24:19.547814 754 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 05:24:19.548089 754 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 05:24:19.549242 754 hybrid_clock.cc:648] HybridClock initialized: now 1786512259549224 us; error 29 us; skew 500 ppm
I20260812 05:24:19.550568 754 webserver.cc:460] Webserver started at http://127.31.100.62:46675/ using document root <none> and password file <none>
I20260812 05:24:19.550789 754 fs_manager.cc:359] Metadata directory not provided
I20260812 05:24:19.550838 754 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 05:24:19.550972 754 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 05:24:19.552000 754 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786512017613366-32144-0/minicluster-data/master-0/data/instance:
uuid: "8813b7b835e04bb9ae74b0382c4f1ba6"
format_stamp: "Formatted at 2026-08-12 05:24:19 on dist-test-slave-42z9"
server_key: "c41c5da005663364f84e0641594b68eb"
server_key_iv: "73f9f9aefa22087bd4e305b2a8c684c5"
server_key_version: "encryptionkey@0"
I20260812 05:24:19.552325 754 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786512017613366-32144-0/minicluster-data/master-0/wal/instance:
uuid: "8813b7b835e04bb9ae74b0382c4f1ba6"
format_stamp: "Formatted at 2026-08-12 05:24:19 on dist-test-slave-42z9"
server_key: "c41c5da005663364f84e0641594b68eb"
server_key_iv: "73f9f9aefa22087bd4e305b2a8c684c5"
server_key_version: "encryptionkey@0"
I20260812 05:24:19.553575 754 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260812 05:24:19.554250 771 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 05:24:19.554415 754 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260812 05:24:19.554503 754 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786512017613366-32144-0/minicluster-data/master-0/data,/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786512017613366-32144-0/minicluster-data/master-0/wal
uuid: "8813b7b835e04bb9ae74b0382c4f1ba6"
format_stamp: "Formatted at 2026-08-12 05:24:19 on dist-test-slave-42z9"
server_key: "c41c5da005663364f84e0641594b68eb"
server_key_iv: "73f9f9aefa22087bd4e305b2a8c684c5"
server_key_version: "encryptionkey@0"
I20260812 05:24:19.554570 754 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786512017613366-32144-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786512017613366-32144-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786512017613366-32144-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 05:24:19.570307 754 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 05:24:19.570638 754 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 05:24:19.570770 754 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 05:24:19.574774 754 rpc_server.cc:307] RPC server started. Bound to: 127.31.100.62:34425
I20260812 05:24:19.574818 855 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.100.62:34425 every 8 connection(s)
I20260812 05:24:19.575140 754 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786512017613366-32144-0/minicluster-data/master-0/data/info.pb
I20260812 05:24:19.576115 859 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 05:24:19.578392 859 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8813b7b835e04bb9ae74b0382c4f1ba6: Bootstrap starting.
I20260812 05:24:19.578929 859 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8813b7b835e04bb9ae74b0382c4f1ba6: Neither blocks nor log segments found. Creating new log.
I20260812 05:24:19.579176 859 log.cc:824] T 00000000000000000000000000000000 P 8813b7b835e04bb9ae74b0382c4f1ba6: Log is configured to *not* fsync() on all Append() calls
I20260812 05:24:19.579821 859 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8813b7b835e04bb9ae74b0382c4f1ba6: No bootstrap required, opened a new log
I20260812 05:24:19.581410 32144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0waFvT/build/release/bin/kudu as pid 754
I20260812 05:24:19.581542 32144 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786512017613366-32144-0/minicluster-data/master-0/wal/instance
I20260812 05:24:19.581509 859 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8813b7b835e04bb9ae74b0382c4f1ba6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8813b7b835e04bb9ae74b0382c4f1ba6" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 34425 } }
I20260812 05:24:19.581709 859 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8813b7b835e04bb9ae74b0382c4f1ba6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 05:24:19.581749 32144 external_mini_cluster.cc:1428] Setting key ee36778a2f4c194ed2642c6b736142c1
I20260812 05:24:19.581753 859 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8813b7b835e04bb9ae74b0382c4f1ba6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8813b7b835e04bb9ae74b0382c4f1ba6, State: Initialized, Role: FOLLOWER
I20260812 05:24:19.581894 859 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8813b7b835e04bb9ae74b0382c4f1ba6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8813b7b835e04bb9ae74b0382c4f1ba6" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 34425 } }
I20260812 05:24:19.581974 859 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8813b7b835e04bb9ae74b0382c4f1ba6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 05:24:19.581997 859 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8813b7b835e04bb9ae74b0382c4f1ba6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 05:24:19.582039 859 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8813b7b835e04bb9ae74b0382c4f1ba6 [term 0 FOLLOWER]: Advancing to term 1
I20260812 05:24:19.582656 859 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8813b7b835e04bb9ae74b0382c4f1ba6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8813b7b835e04bb9ae74b0382c4f1ba6" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 34425 } }
I20260812 05:24:19.582749 859 leader_election.cc:302] T 00000000000000000000000000000000 P 8813b7b835e04bb9ae74b0382c4f1ba6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8813b7b835e04bb9ae74b0382c4f1ba6; no voters:
I20260812 05:24:19.582892 859 leader_election.cc:288] T 00000000000000000000000000000000 P 8813b7b835e04bb9ae74b0382c4f1ba6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 05:24:19.582954 864 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8813b7b835e04bb9ae74b0382c4f1ba6 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 05:24:19.583097 864 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8813b7b835e04bb9ae74b0382c4f1ba6 [term 1 LEADER]: Becoming Leader. State: Replica: 8813b7b835e04bb9ae74b0382c4f1ba6, State: Running, Role: LEADER
I20260812 05:24:19.583143 859 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8813b7b835e04bb9ae74b0382c4f1ba6 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 05:24:19.583220 864 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8813b7b835e04bb9ae74b0382c4f1ba6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8813b7b835e04bb9ae74b0382c4f1ba6" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 34425 } }
I20260812 05:24:19.583595 865 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8813b7b835e04bb9ae74b0382c4f1ba6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8813b7b835e04bb9ae74b0382c4f1ba6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8813b7b835e04bb9ae74b0382c4f1ba6" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 34425 } } }
I20260812 05:24:19.583679 865 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8813b7b835e04bb9ae74b0382c4f1ba6 [sys.catalog]: This master's current role is: LEADER
I20260812 05:24:19.583809 865 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8813b7b835e04bb9ae74b0382c4f1ba6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8813b7b835e04bb9ae74b0382c4f1ba6. Latest consensus state: current_term: 1 leader_uuid: "8813b7b835e04bb9ae74b0382c4f1ba6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8813b7b835e04bb9ae74b0382c4f1ba6" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 34425 } } }
I20260812 05:24:19.583858 865 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8813b7b835e04bb9ae74b0382c4f1ba6 [sys.catalog]: This master's current role is: LEADER
I20260812 05:24:19.583966 876 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 05:24:19.584707 876 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 05:24:19.586308 876 catalog_manager.cc:1347] Generated new cluster ID: 147e89a592c84e16bdf4e3fbc9e941ff
I20260812 05:24:19.586364 876 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 05:24:19.588173 863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:24:19.582273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36128 (local address 127.31.100.62:34425)
0812 05:24:19.582498 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0812 05:24:19.582506 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 05:24:19.582531 (+ 25us) server_negotiation.cc:407] Connection header received
0812 05:24:19.582642 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:24:19.582648 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:24:19.582711 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:24:19.582847 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:24:19.583204 (+ 357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:24:19.585056 (+ 1852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:24:19.586169 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:24:19.586686 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:24:19.586802 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:24:19.587474 (+ 672us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 05:24:19.587478 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 05:24:19.587479 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 05:24:19.587657 (+ 178us) server_negotiation.cc:922] calling sasl_server_start()
0812 05:24:19.587691 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 05:24:19.587702 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 05:24:19.587758 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0812 05:24:19.587843 (+ 85us) server_negotiation.cc:299] Negotiation successful
0812 05:24:19.588009 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":25,"server-negotiator.queue_time_us":143,"thread_start_us":92,"threads_started":1}
I20260812 05:24:19.599635 876 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 05:24:19.600304 876 catalog_manager.cc:1504] Loading token signing keys...
I20260812 05:24:19.603433 876 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8813b7b835e04bb9ae74b0382c4f1ba6: Generated new TSK 0
I20260812 05:24:19.603611 876 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 05:24:19.606251 32144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0waFvT/build/release/bin/kudu
/tmp/dist-test-task0waFvT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786512017613366-32144-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786512017613366-32144-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786512017613366-32144-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786512017613366-32144-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.100.1:0
--local_ip_for_outbound_sockets=127.31.100.1
--webserver_interface=127.31.100.1
--webserver_port=0
--tserver_master_addrs=127.31.100.62:34425
--builtin_ntp_servers=127.31.100.20:39481
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260812 05:24:19.689291 897 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 05:24:19.689559 897 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 05:24:19.689586 897 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 05:24:19.690811 897 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 05:24:19.690860 897 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 05:24:19.690886 897 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.100.1
I20260812 05:24:19.692304 897 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.100.20:39481
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786512017613366-32144-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786512017613366-32144-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.100.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786512017613366-32144-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.100.1
--webserver_port=0
--tserver_master_addrs=127.31.100.62:34425
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.897
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.100.1
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786512017613366-32144-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 04:20:34 UTC on ccycloud.quasar-phqdqp.root.comops.site
build id 4581871
I20260812 05:24:19.692552 897 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 05:24:19.692876 897 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 05:24:19.698971 909 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 05:24:19.698948 914 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 05:24:19.698937 912 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 05:24:19.699375 897 server_base.cc:1034] running on GCE node
I20260812 05:24:19.699587 897 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 05:24:19.699841 897 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 05:24:19.700994 897 hybrid_clock.cc:648] HybridClock initialized: now 1786512259700983 us; error 27 us; skew 500 ppm
I20260812 05:24:19.702309 897 webserver.cc:460] Webserver started at http://127.31.100.1:45137/ using document root <none> and password file <none>
I20260812 05:24:19.702562 897 fs_manager.cc:359] Metadata directory not provided
I20260812 05:24:19.702622 897 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 05:24:19.702759 897 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 05:24:19.703835 897 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786512017613366-32144-0/minicluster-data/ts-0/data/instance:
uuid: "02ae7d5ca6e1473b8b0afccf84aca901"
format_stamp: "Formatted at 2026-08-12 05:24:19 on dist-test-slave-42z9"
server_key: "0f3d43b4e38556d66d9ae6bc3e05511f"
server_key_iv: "3d98fc9989616a005114e879e34a25dc"
server_key_version: "encryptionkey@0"
I20260812 05:24:19.704173 897 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786512017613366-32144-0/minicluster-data/ts-0/wal/instance:
uuid: "02ae7d5ca6e1473b8b0afccf84aca901"
format_stamp: "Formatted at 2026-08-12 05:24:19 on dist-test-slave-42z9"
server_key: "0f3d43b4e38556d66d9ae6bc3e05511f"
server_key_iv: "3d98fc9989616a005114e879e34a25dc"
server_key_version: "encryptionkey@0"
I20260812 05:24:19.705523 897 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260812 05:24:19.706367 923 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 05:24:19.706552 897 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 05:24:19.706647 897 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786512017613366-32144-0/minicluster-data/ts-0/data,/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786512017613366-32144-0/minicluster-data/ts-0/wal
uuid: "02ae7d5ca6e1473b8b0afccf84aca901"
format_stamp: "Formatted at 2026-08-12 05:24:19 on dist-test-slave-42z9"
server_key: "0f3d43b4e38556d66d9ae6bc3e05511f"
server_key_iv: "3d98fc9989616a005114e879e34a25dc"
server_key_version: "encryptionkey@0"
I20260812 05:24:19.706717 897 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786512017613366-32144-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786512017613366-32144-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786512017613366-32144-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 05:24:19.735033 897 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 05:24:19.735368 897 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 05:24:19.735502 897 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 05:24:19.735741 897 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 05:24:19.736076 897 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 05:24:19.736114 897 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 05:24:19.736145 897 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 05:24:19.736168 897 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 05:24:19.741859 897 rpc_server.cc:307] RPC server started. Bound to: 127.31.100.1:35073
I20260812 05:24:19.741921 1116 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.100.1:35073 every 8 connection(s)
I20260812 05:24:19.742228 897 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786512017613366-32144-0/minicluster-data/ts-0/data/info.pb
I20260812 05:24:19.742473 32144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0waFvT/build/release/bin/kudu as pid 897
I20260812 05:24:19.742599 32144 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786512017613366-32144-0/minicluster-data/ts-0/wal/instance
I20260812 05:24:19.742817 32144 external_mini_cluster.cc:1428] Setting key 2517699ec9af7cfc47b0cc96142f7b35
I20260812 05:24:19.743817 32144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0waFvT/build/release/bin/kudu
/tmp/dist-test-task0waFvT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786512017613366-32144-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786512017613366-32144-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786512017613366-32144-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786512017613366-32144-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.100.2:0
--local_ip_for_outbound_sockets=127.31.100.2
--webserver_interface=127.31.100.2
--webserver_port=0
--tserver_master_addrs=127.31.100.62:34425
--builtin_ntp_servers=127.31.100.20:39481
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260812 05:24:19.747871 863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:24:19.742970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.100.1:49695 (local address 127.31.100.62:34425)
0812 05:24:19.743021 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0812 05:24:19.743023 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 05:24:19.743295 (+ 272us) server_negotiation.cc:407] Connection header received
0812 05:24:19.743784 (+ 489us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:24:19.743786 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:24:19.743801 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:24:19.743871 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:24:19.744579 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:24:19.745078 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:24:19.746461 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:24:19.746740 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:24:19.746816 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:24:19.747562 (+ 746us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 05:24:19.747564 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 05:24:19.747564 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 05:24:19.747570 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0812 05:24:19.747587 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 05:24:19.747592 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 05:24:19.747653 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0812 05:24:19.747752 (+ 99us) server_negotiation.cc:299] Negotiation successful
0812 05:24:19.747788 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260812 05:24:19.747984 1119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:24:19.742969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.62:34425 (local address 127.31.100.1:49695)
0812 05:24:19.743176 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0812 05:24:19.743216 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0812 05:24:19.743685 (+ 469us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:24:19.743945 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:24:19.743949 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:24:19.743960 (+ 11us) client_negotiation.cc:189] Negotiated authn=SASL
0812 05:24:19.744485 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:24:19.744490 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:24:19.745162 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:24:19.745163 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:24:19.746372 (+ 1209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:24:19.746376 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:24:19.746822 (+ 446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:24:19.746823 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:24:19.746936 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:24:19.747432 (+ 496us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 05:24:19.747433 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0812 05:24:19.747460 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 05:24:19.747481 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 05:24:19.747684 (+ 203us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 05:24:19.747690 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 05:24:19.747692 (+ 2us) client_negotiation.cc:769] Sending connection context
0812 05:24:19.747739 (+ 47us) client_negotiation.cc:240] Negotiation successful
0812 05:24:19.747819 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":93,"threads_started":1}
I20260812 05:24:19.748440 1117 heartbeater.cc:347] Connected to a master server at 127.31.100.62:34425
I20260812 05:24:19.748543 1117 heartbeater.cc:464] Registering TS with master...
I20260812 05:24:19.748731 1117 heartbeater.cc:511] Master 127.31.100.62:34425 requested a full tablet report, sending...
I20260812 05:24:19.749228 798 ts_manager.cc:194] Registered new tserver with Master: 02ae7d5ca6e1473b8b0afccf84aca901 (127.31.100.1:35073)
I20260812 05:24:19.749869 798 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.31.100.1:49695
W20260812 05:24:19.828481 1120 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 05:24:19.828689 1120 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 05:24:19.828711 1120 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 05:24:19.829874 1120 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 05:24:19.829914 1120 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 05:24:19.829937 1120 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.100.2
I20260812 05:24:19.831287 1120 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.100.20:39481
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786512017613366-32144-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786512017613366-32144-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.100.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786512017613366-32144-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.100.2
--webserver_port=0
--tserver_master_addrs=127.31.100.62:34425
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1120
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.100.2
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786512017613366-32144-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 04:20:34 UTC on ccycloud.quasar-phqdqp.root.comops.site
build id 4581871
I20260812 05:24:19.831548 1120 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 05:24:19.831841 1120 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 05:24:19.836733 1132 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 05:24:19.836799 1130 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 05:24:19.836733 1129 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 05:24:19.836937 1120 server_base.cc:1034] running on GCE node
I20260812 05:24:19.837165 1120 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 05:24:19.837411 1120 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 05:24:19.838577 1120 hybrid_clock.cc:648] HybridClock initialized: now 1786512259838560 us; error 34 us; skew 500 ppm
I20260812 05:24:19.839859 1120 webserver.cc:460] Webserver started at http://127.31.100.2:40105/ using document root <none> and password file <none>
I20260812 05:24:19.840095 1120 fs_manager.cc:359] Metadata directory not provided
I20260812 05:24:19.840143 1120 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 05:24:19.840269 1120 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 05:24:19.841317 1120 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786512017613366-32144-0/minicluster-data/ts-1/data/instance:
uuid: "4ef447bad3514908ab6c0b8e38159b5b"
format_stamp: "Formatted at 2026-08-12 05:24:19 on dist-test-slave-42z9"
server_key: "884b049312655d5129037bab06869f3d"
server_key_iv: "d967d34288d4e3b076ff2f1ba38b0f02"
server_key_version: "encryptionkey@0"
I20260812 05:24:19.841708 1120 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786512017613366-32144-0/minicluster-data/ts-1/wal/instance:
uuid: "4ef447bad3514908ab6c0b8e38159b5b"
format_stamp: "Formatted at 2026-08-12 05:24:19 on dist-test-slave-42z9"
server_key: "884b049312655d5129037bab06869f3d"
server_key_iv: "d967d34288d4e3b076ff2f1ba38b0f02"
server_key_version: "encryptionkey@0"
I20260812 05:24:19.842942 1120 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260812 05:24:19.843803 1143 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 05:24:19.843941 1120 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 05:24:19.844029 1120 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786512017613366-32144-0/minicluster-data/ts-1/data,/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786512017613366-32144-0/minicluster-data/ts-1/wal
uuid: "4ef447bad3514908ab6c0b8e38159b5b"
format_stamp: "Formatted at 2026-08-12 05:24:19 on dist-test-slave-42z9"
server_key: "884b049312655d5129037bab06869f3d"
server_key_iv: "d967d34288d4e3b076ff2f1ba38b0f02"
server_key_version: "encryptionkey@0"
I20260812 05:24:19.844096 1120 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786512017613366-32144-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786512017613366-32144-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786512017613366-32144-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 05:24:19.859723 1120 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 05:24:19.859982 1120 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 05:24:19.860080 1120 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 05:24:19.860257 1120 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 05:24:19.860554 1120 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 05:24:19.860586 1120 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 05:24:19.860606 1120 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 05:24:19.860625 1120 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 05:24:19.865876 1120 rpc_server.cc:307] RPC server started. Bound to: 127.31.100.2:33013
I20260812 05:24:19.865937 1333 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.100.2:33013 every 8 connection(s)
I20260812 05:24:19.866238 1120 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786512017613366-32144-0/minicluster-data/ts-1/data/info.pb
I20260812 05:24:19.869948 32144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0waFvT/build/release/bin/kudu as pid 1120
I20260812 05:24:19.870076 32144 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786512017613366-32144-0/minicluster-data/ts-1/wal/instance
I20260812 05:24:19.870267 32144 external_mini_cluster.cc:1428] Setting key a2612eb9384f777b032951812cacb517
I20260812 05:24:19.871414 32144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0waFvT/build/release/bin/kudu
/tmp/dist-test-task0waFvT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786512017613366-32144-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786512017613366-32144-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786512017613366-32144-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786512017613366-32144-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.100.3:0
--local_ip_for_outbound_sockets=127.31.100.3
--webserver_interface=127.31.100.3
--webserver_port=0
--tserver_master_addrs=127.31.100.62:34425
--builtin_ntp_servers=127.31.100.20:39481
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260812 05:24:19.871512 863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:24:19.866906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.100.2:50467 (local address 127.31.100.62:34425)
0812 05:24:19.866967 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0812 05:24:19.866969 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 05:24:19.867182 (+ 213us) server_negotiation.cc:407] Connection header received
0812 05:24:19.867702 (+ 520us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:24:19.867703 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:24:19.867716 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:24:19.867770 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:24:19.868427 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:24:19.868927 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:24:19.870028 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:24:19.870294 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:24:19.870370 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:24:19.871251 (+ 881us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 05:24:19.871252 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 05:24:19.871252 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 05:24:19.871256 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0812 05:24:19.871273 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 05:24:19.871277 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 05:24:19.871327 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0812 05:24:19.871400 (+ 73us) server_negotiation.cc:299] Negotiation successful
0812 05:24:19.871432 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260812 05:24:19.871727 1341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:24:19.866893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.62:34425 (local address 127.31.100.2:50467)
0812 05:24:19.867062 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0812 05:24:19.867095 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0812 05:24:19.867599 (+ 504us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:24:19.867834 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:24:19.867838 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:24:19.867846 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0812 05:24:19.868335 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:24:19.868340 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:24:19.869012 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:24:19.869013 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:24:19.869929 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:24:19.869932 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:24:19.870385 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:24:19.870386 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:24:19.870546 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:24:19.871089 (+ 543us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 05:24:19.871090 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0812 05:24:19.871142 (+ 52us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 05:24:19.871171 (+ 29us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 05:24:19.871333 (+ 162us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 05:24:19.871339 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 05:24:19.871342 (+ 3us) client_negotiation.cc:769] Sending connection context
0812 05:24:19.871404 (+ 62us) client_negotiation.cc:240] Negotiation successful
0812 05:24:19.871508 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":76,"threads_started":1}
I20260812 05:24:19.872161 1337 heartbeater.cc:347] Connected to a master server at 127.31.100.62:34425
I20260812 05:24:19.872265 1337 heartbeater.cc:464] Registering TS with master...
I20260812 05:24:19.872458 1337 heartbeater.cc:511] Master 127.31.100.62:34425 requested a full tablet report, sending...
I20260812 05:24:19.872823 798 ts_manager.cc:194] Registered new tserver with Master: 4ef447bad3514908ab6c0b8e38159b5b (127.31.100.2:33013)
I20260812 05:24:19.873266 798 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.31.100.2:50467
W20260812 05:24:19.954474 1342 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 05:24:19.954672 1342 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 05:24:19.954692 1342 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 05:24:19.955804 1342 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 05:24:19.955844 1342 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 05:24:19.955871 1342 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.100.3
I20260812 05:24:19.957263 1342 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.100.20:39481
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786512017613366-32144-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786512017613366-32144-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.100.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786512017613366-32144-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.100.3
--webserver_port=0
--tserver_master_addrs=127.31.100.62:34425
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1342
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.100.3
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786512017613366-32144-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 04:20:34 UTC on ccycloud.quasar-phqdqp.root.comops.site
build id 4581871
I20260812 05:24:19.957538 1342 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 05:24:19.957823 1342 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 05:24:19.962613 1350 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 05:24:19.962613 1355 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 05:24:19.962741 1342 server_base.cc:1034] running on GCE node
W20260812 05:24:19.962642 1352 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 05:24:19.963083 1342 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 05:24:19.963344 1342 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 05:24:19.964484 1342 hybrid_clock.cc:648] HybridClock initialized: now 1786512259964472 us; error 46 us; skew 500 ppm
I20260812 05:24:19.965766 1342 webserver.cc:460] Webserver started at http://127.31.100.3:37795/ using document root <none> and password file <none>
I20260812 05:24:19.966017 1342 fs_manager.cc:359] Metadata directory not provided
I20260812 05:24:19.966068 1342 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 05:24:19.966219 1342 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 05:24:19.967505 1342 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786512017613366-32144-0/minicluster-data/ts-2/data/instance:
uuid: "da01691963664f67b1c3a3b3724a935f"
format_stamp: "Formatted at 2026-08-12 05:24:19 on dist-test-slave-42z9"
server_key: "c79f45438a085207d187a2634acc108e"
server_key_iv: "6a7731b22d89510ef40ead7c1cd08f00"
server_key_version: "encryptionkey@0"
I20260812 05:24:19.967969 1342 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786512017613366-32144-0/minicluster-data/ts-2/wal/instance:
uuid: "da01691963664f67b1c3a3b3724a935f"
format_stamp: "Formatted at 2026-08-12 05:24:19 on dist-test-slave-42z9"
server_key: "c79f45438a085207d187a2634acc108e"
server_key_iv: "6a7731b22d89510ef40ead7c1cd08f00"
server_key_version: "encryptionkey@0"
I20260812 05:24:19.969339 1342 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260812 05:24:19.970148 1370 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 05:24:19.970295 1342 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260812 05:24:19.970372 1342 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786512017613366-32144-0/minicluster-data/ts-2/data,/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786512017613366-32144-0/minicluster-data/ts-2/wal
uuid: "da01691963664f67b1c3a3b3724a935f"
format_stamp: "Formatted at 2026-08-12 05:24:19 on dist-test-slave-42z9"
server_key: "c79f45438a085207d187a2634acc108e"
server_key_iv: "6a7731b22d89510ef40ead7c1cd08f00"
server_key_version: "encryptionkey@0"
I20260812 05:24:19.970433 1342 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786512017613366-32144-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786512017613366-32144-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786512017613366-32144-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 05:24:19.989542 1342 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 05:24:19.989853 1342 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 05:24:19.989984 1342 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 05:24:19.990221 1342 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 05:24:19.990535 1342 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 05:24:19.990571 1342 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 05:24:19.990599 1342 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 05:24:19.990617 1342 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 05:24:19.995945 1342 rpc_server.cc:307] RPC server started. Bound to: 127.31.100.3:37423
I20260812 05:24:19.996009 1560 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.100.3:37423 every 8 connection(s)
I20260812 05:24:19.996317 1342 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786512017613366-32144-0/minicluster-data/ts-2/data/info.pb
I20260812 05:24:19.997290 32144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0waFvT/build/release/bin/kudu as pid 1342
I20260812 05:24:19.997404 32144 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786512017613366-32144-0/minicluster-data/ts-2/wal/instance
I20260812 05:24:19.997623 32144 external_mini_cluster.cc:1428] Setting key edb56f69a022782dfbad884960e63aa4
I20260812 05:24:20.002030 863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:24:19.997049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.100.3:33213 (local address 127.31.100.62:34425)
0812 05:24:19.997109 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0812 05:24:19.997111 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 05:24:19.997386 (+ 275us) server_negotiation.cc:407] Connection header received
0812 05:24:19.998104 (+ 718us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:24:19.998105 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:24:19.998117 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:24:19.998156 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:24:19.998972 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:24:19.999488 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:24:20.000677 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:24:20.000892 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:24:20.000946 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:24:20.001767 (+ 821us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 05:24:20.001768 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 05:24:20.001768 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 05:24:20.001773 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0812 05:24:20.001789 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 05:24:20.001793 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 05:24:20.001839 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0812 05:24:20.001919 (+ 80us) server_negotiation.cc:299] Negotiation successful
0812 05:24:20.001950 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260812 05:24:20.002223 1563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:24:19.997014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.62:34425 (local address 127.31.100.3:33213)
0812 05:24:19.997228 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0812 05:24:19.997278 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0812 05:24:19.997918 (+ 640us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:24:19.998345 (+ 427us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:24:19.998349 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:24:19.998358 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0812 05:24:19.998887 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:24:19.998892 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:24:19.999578 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:24:19.999579 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:24:20.000562 (+ 983us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:24:20.000564 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:24:20.000962 (+ 398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:24:20.000964 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:24:20.001091 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:24:20.001633 (+ 542us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 05:24:20.001633 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0812 05:24:20.001662 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 05:24:20.001686 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 05:24:20.001849 (+ 163us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 05:24:20.001854 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 05:24:20.001857 (+ 3us) client_negotiation.cc:769] Sending connection context
0812 05:24:20.001916 (+ 59us) client_negotiation.cc:240] Negotiation successful
0812 05:24:20.002018 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":95,"threads_started":1}
I20260812 05:24:20.002651 1561 heartbeater.cc:347] Connected to a master server at 127.31.100.62:34425
I20260812 05:24:20.002734 1561 heartbeater.cc:464] Registering TS with master...
I20260812 05:24:20.002879 1561 heartbeater.cc:511] Master 127.31.100.62:34425 requested a full tablet report, sending...
I20260812 05:24:20.003180 798 ts_manager.cc:194] Registered new tserver with Master: da01691963664f67b1c3a3b3724a935f (127.31.100.3:37423)
I20260812 05:24:20.003494 798 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.31.100.3:33213
I20260812 05:24:20.009251 32144 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 05:24:20.013217 863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:24:20.010569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53316 (local address 127.31.100.62:34425)
0812 05:24:20.010607 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0812 05:24:20.010609 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 05:24:20.010714 (+ 105us) server_negotiation.cc:407] Connection header received
0812 05:24:20.010773 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:24:20.010774 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:24:20.010785 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:24:20.010829 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:24:20.010972 (+ 143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:24:20.011430 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:24:20.012155 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:24:20.012337 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:24:20.012391 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:24:20.012990 (+ 599us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 05:24:20.012992 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 05:24:20.012992 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 05:24:20.012996 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0812 05:24:20.013010 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 05:24:20.013014 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 05:24:20.013058 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0812 05:24:20.013124 (+ 66us) server_negotiation.cc:299] Negotiation successful
0812 05:24:20.013146 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260812 05:24:20.014525 798 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53316:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 05:24:20.016129 798 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 05:24:20.022119 1577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:24:20.018410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46214 (local address 127.31.100.2:33013)
0812 05:24:20.018587 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0812 05:24:20.018592 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 05:24:20.018601 (+ 9us) server_negotiation.cc:407] Connection header received
0812 05:24:20.018636 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:24:20.018638 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:24:20.018653 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:24:20.018717 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 05:24:20.018875 (+ 158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:24:20.019784 (+ 909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:24:20.021165 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:24:20.021741 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:24:20.021808 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:24:20.021904 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0812 05:24:20.021989 (+ 85us) server_negotiation.cc:299] Negotiation successful
0812 05:24:20.022031 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":88,"threads_started":1}
I20260812 05:24:20.022329 1575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:24:20.018112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.3:37423 (local address 127.0.0.1:53074)
0812 05:24:20.018295 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0812 05:24:20.018332 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0812 05:24:20.018394 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:24:20.018831 (+ 437us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:24:20.018832 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:24:20.018835 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 05:24:20.018881 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:24:20.018882 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:24:20.020101 (+ 1219us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:24:20.020102 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:24:20.021171 (+ 1069us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:24:20.021173 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:24:20.022125 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:24:20.022126 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:24:20.022182 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:24:20.022185 (+ 3us) client_negotiation.cc:769] Sending connection context
0812 05:24:20.022230 (+ 45us) client_negotiation.cc:240] Negotiation successful
0812 05:24:20.022267 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162,"mutex_wait_us":23,"thread_start_us":76,"threads_started":1}
I20260812 05:24:20.022362 1574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:24:20.018111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.1:35073 (local address 127.0.0.1:45022)
0812 05:24:20.018295 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0812 05:24:20.018332 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0812 05:24:20.018394 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:24:20.018818 (+ 424us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:24:20.018819 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:24:20.018821 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 05:24:20.018852 (+ 31us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:24:20.018854 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:24:20.020145 (+ 1291us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:24:20.020146 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:24:20.020920 (+ 774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:24:20.020923 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:24:20.021870 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:24:20.021871 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:24:20.021938 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:24:20.021943 (+ 5us) client_negotiation.cc:769] Sending connection context
0812 05:24:20.021974 (+ 31us) client_negotiation.cc:240] Negotiation successful
0812 05:24:20.022057 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":76,"threads_started":1}
I20260812 05:24:20.022428 1578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:24:20.018439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45022 (local address 127.31.100.1:35073)
0812 05:24:20.018620 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0812 05:24:20.018624 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 05:24:20.018632 (+ 8us) server_negotiation.cc:407] Connection header received
0812 05:24:20.018668 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:24:20.018670 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:24:20.018684 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:24:20.018798 (+ 114us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 05:24:20.018907 (+ 109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:24:20.020047 (+ 1140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:24:20.021008 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:24:20.021764 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:24:20.022187 (+ 423us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:24:20.022284 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0812 05:24:20.022310 (+ 26us) server_negotiation.cc:299] Negotiation successful
0812 05:24:20.022349 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":67,"threads_started":1}
I20260812 05:24:20.022132 1571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:24:20.018134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.2:33013 (local address 127.0.0.1:46214)
0812 05:24:20.018502 (+ 368us) negotiation.cc:107] Waiting for socket to connect
0812 05:24:20.018510 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0812 05:24:20.018545 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:24:20.018728 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:24:20.018732 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:24:20.018733 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 05:24:20.018796 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:24:20.018797 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:24:20.019862 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:24:20.019863 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:24:20.021068 (+ 1205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:24:20.021071 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:24:20.021818 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:24:20.021819 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:24:20.021896 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:24:20.021909 (+ 13us) client_negotiation.cc:769] Sending connection context
0812 05:24:20.021977 (+ 68us) client_negotiation.cc:240] Negotiation successful
0812 05:24:20.022057 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":358,"mutex_wait_us":36,"thread_start_us":100,"threads_started":1}
I20260812 05:24:20.022774 1576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:24:20.018279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53074 (local address 127.31.100.3:37423)
0812 05:24:20.018614 (+ 335us) server_negotiation.cc:206] Beginning negotiation
0812 05:24:20.018619 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 05:24:20.018629 (+ 10us) server_negotiation.cc:407] Connection header received
0812 05:24:20.018667 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:24:20.018669 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:24:20.018684 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:24:20.018767 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 05:24:20.018957 (+ 190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:24:20.020016 (+ 1059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:24:20.021245 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:24:20.021950 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:24:20.022008 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:24:20.022090 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0812 05:24:20.022438 (+ 348us) server_negotiation.cc:299] Negotiation successful
0812 05:24:20.022694 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":298,"thread_start_us":77,"threads_started":1}
I20260812 05:24:20.023175 1223 tablet_service.cc:1467] Processing CreateTablet for tablet 9b2018442bf0441f9bbe1c7e36dac068 (DEFAULT_TABLE table=test-table [id=59f462c5e64942c4986acb3f9c9868f3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 05:24:20.023156 1013 tablet_service.cc:1467] Processing CreateTablet for tablet 9b2018442bf0441f9bbe1c7e36dac068 (DEFAULT_TABLE table=test-table [id=59f462c5e64942c4986acb3f9c9868f3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 05:24:20.023537 1223 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b2018442bf0441f9bbe1c7e36dac068. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 05:24:20.023537 1013 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b2018442bf0441f9bbe1c7e36dac068. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 05:24:20.026120 1579 tablet_bootstrap.cc:492] T 9b2018442bf0441f9bbe1c7e36dac068 P 4ef447bad3514908ab6c0b8e38159b5b: Bootstrap starting.
I20260812 05:24:20.026399 1580 tablet_bootstrap.cc:492] T 9b2018442bf0441f9bbe1c7e36dac068 P 02ae7d5ca6e1473b8b0afccf84aca901: Bootstrap starting.
I20260812 05:24:20.026628 1450 tablet_service.cc:1467] Processing CreateTablet for tablet 9b2018442bf0441f9bbe1c7e36dac068 (DEFAULT_TABLE table=test-table [id=59f462c5e64942c4986acb3f9c9868f3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 05:24:20.026981 1450 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b2018442bf0441f9bbe1c7e36dac068. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 05:24:20.027048 1580 tablet_bootstrap.cc:654] T 9b2018442bf0441f9bbe1c7e36dac068 P 02ae7d5ca6e1473b8b0afccf84aca901: Neither blocks nor log segments found. Creating new log.
I20260812 05:24:20.027127 1579 tablet_bootstrap.cc:654] T 9b2018442bf0441f9bbe1c7e36dac068 P 4ef447bad3514908ab6c0b8e38159b5b: Neither blocks nor log segments found. Creating new log.
I20260812 05:24:20.027360 1580 log.cc:824] T 9b2018442bf0441f9bbe1c7e36dac068 P 02ae7d5ca6e1473b8b0afccf84aca901: Log is configured to *not* fsync() on all Append() calls
I20260812 05:24:20.027441 1579 log.cc:824] T 9b2018442bf0441f9bbe1c7e36dac068 P 4ef447bad3514908ab6c0b8e38159b5b: Log is configured to *not* fsync() on all Append() calls
I20260812 05:24:20.028028 1580 tablet_bootstrap.cc:492] T 9b2018442bf0441f9bbe1c7e36dac068 P 02ae7d5ca6e1473b8b0afccf84aca901: No bootstrap required, opened a new log
I20260812 05:24:20.028093 1580 ts_tablet_manager.cc:1397] T 9b2018442bf0441f9bbe1c7e36dac068 P 02ae7d5ca6e1473b8b0afccf84aca901: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260812 05:24:20.029670 1584 tablet_bootstrap.cc:492] T 9b2018442bf0441f9bbe1c7e36dac068 P da01691963664f67b1c3a3b3724a935f: Bootstrap starting.
I20260812 05:24:20.030108 1580 raft_consensus.cc:348] T 9b2018442bf0441f9bbe1c7e36dac068 P 02ae7d5ca6e1473b8b0afccf84aca901 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da01691963664f67b1c3a3b3724a935f" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 37423 } } peers { permanent_uuid: "02ae7d5ca6e1473b8b0afccf84aca901" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 35073 } } peers { permanent_uuid: "4ef447bad3514908ab6c0b8e38159b5b" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 33013 } }
I20260812 05:24:20.030241 1580 raft_consensus.cc:374] T 9b2018442bf0441f9bbe1c7e36dac068 P 02ae7d5ca6e1473b8b0afccf84aca901 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 05:24:20.030262 1580 raft_consensus.cc:729] T 9b2018442bf0441f9bbe1c7e36dac068 P 02ae7d5ca6e1473b8b0afccf84aca901 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02ae7d5ca6e1473b8b0afccf84aca901, State: Initialized, Role: FOLLOWER
I20260812 05:24:20.030272 1584 tablet_bootstrap.cc:654] T 9b2018442bf0441f9bbe1c7e36dac068 P da01691963664f67b1c3a3b3724a935f: Neither blocks nor log segments found. Creating new log.
I20260812 05:24:20.030361 1580 consensus_queue.cc:260] T 9b2018442bf0441f9bbe1c7e36dac068 P 02ae7d5ca6e1473b8b0afccf84aca901 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da01691963664f67b1c3a3b3724a935f" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 37423 } } peers { permanent_uuid: "02ae7d5ca6e1473b8b0afccf84aca901" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 35073 } } peers { permanent_uuid: "4ef447bad3514908ab6c0b8e38159b5b" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 33013 } }
I20260812 05:24:20.030573 1584 log.cc:824] T 9b2018442bf0441f9bbe1c7e36dac068 P da01691963664f67b1c3a3b3724a935f: Log is configured to *not* fsync() on all Append() calls
I20260812 05:24:20.030613 1580 ts_tablet_manager.cc:1428] T 9b2018442bf0441f9bbe1c7e36dac068 P 02ae7d5ca6e1473b8b0afccf84aca901: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260812 05:24:20.030678 1117 heartbeater.cc:503] Master 127.31.100.62:34425 was elected leader, sending a full tablet report...
I20260812 05:24:20.031154 1579 tablet_bootstrap.cc:492] T 9b2018442bf0441f9bbe1c7e36dac068 P 4ef447bad3514908ab6c0b8e38159b5b: No bootstrap required, opened a new log
I20260812 05:24:20.031216 1579 ts_tablet_manager.cc:1397] T 9b2018442bf0441f9bbe1c7e36dac068 P 4ef447bad3514908ab6c0b8e38159b5b: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20260812 05:24:20.031301 1584 tablet_bootstrap.cc:492] T 9b2018442bf0441f9bbe1c7e36dac068 P da01691963664f67b1c3a3b3724a935f: No bootstrap required, opened a new log
I20260812 05:24:20.031375 1584 ts_tablet_manager.cc:1397] T 9b2018442bf0441f9bbe1c7e36dac068 P da01691963664f67b1c3a3b3724a935f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260812 05:24:20.033093 1579 raft_consensus.cc:348] T 9b2018442bf0441f9bbe1c7e36dac068 P 4ef447bad3514908ab6c0b8e38159b5b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da01691963664f67b1c3a3b3724a935f" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 37423 } } peers { permanent_uuid: "02ae7d5ca6e1473b8b0afccf84aca901" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 35073 } } peers { permanent_uuid: "4ef447bad3514908ab6c0b8e38159b5b" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 33013 } }
I20260812 05:24:20.033221 1579 raft_consensus.cc:374] T 9b2018442bf0441f9bbe1c7e36dac068 P 4ef447bad3514908ab6c0b8e38159b5b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 05:24:20.033252 1579 raft_consensus.cc:729] T 9b2018442bf0441f9bbe1c7e36dac068 P 4ef447bad3514908ab6c0b8e38159b5b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ef447bad3514908ab6c0b8e38159b5b, State: Initialized, Role: FOLLOWER
I20260812 05:24:20.033361 1579 consensus_queue.cc:260] T 9b2018442bf0441f9bbe1c7e36dac068 P 4ef447bad3514908ab6c0b8e38159b5b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da01691963664f67b1c3a3b3724a935f" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 37423 } } peers { permanent_uuid: "02ae7d5ca6e1473b8b0afccf84aca901" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 35073 } } peers { permanent_uuid: "4ef447bad3514908ab6c0b8e38159b5b" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 33013 } }
I20260812 05:24:20.033288 1584 raft_consensus.cc:348] T 9b2018442bf0441f9bbe1c7e36dac068 P da01691963664f67b1c3a3b3724a935f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da01691963664f67b1c3a3b3724a935f" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 37423 } } peers { permanent_uuid: "02ae7d5ca6e1473b8b0afccf84aca901" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 35073 } } peers { permanent_uuid: "4ef447bad3514908ab6c0b8e38159b5b" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 33013 } }
I20260812 05:24:20.033473 1584 raft_consensus.cc:374] T 9b2018442bf0441f9bbe1c7e36dac068 P da01691963664f67b1c3a3b3724a935f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 05:24:20.033522 1584 raft_consensus.cc:729] T 9b2018442bf0441f9bbe1c7e36dac068 P da01691963664f67b1c3a3b3724a935f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: da01691963664f67b1c3a3b3724a935f, State: Initialized, Role: FOLLOWER
I20260812 05:24:20.033634 1579 ts_tablet_manager.cc:1428] T 9b2018442bf0441f9bbe1c7e36dac068 P 4ef447bad3514908ab6c0b8e38159b5b: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260812 05:24:20.033630 1584 consensus_queue.cc:260] T 9b2018442bf0441f9bbe1c7e36dac068 P da01691963664f67b1c3a3b3724a935f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da01691963664f67b1c3a3b3724a935f" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 37423 } } peers { permanent_uuid: "02ae7d5ca6e1473b8b0afccf84aca901" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 35073 } } peers { permanent_uuid: "4ef447bad3514908ab6c0b8e38159b5b" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 33013 } }
I20260812 05:24:20.033761 1337 heartbeater.cc:503] Master 127.31.100.62:34425 was elected leader, sending a full tablet report...
I20260812 05:24:20.033866 1584 ts_tablet_manager.cc:1428] T 9b2018442bf0441f9bbe1c7e36dac068 P da01691963664f67b1c3a3b3724a935f: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260812 05:24:20.033989 1561 heartbeater.cc:503] Master 127.31.100.62:34425 was elected leader, sending a full tablet report...
W20260812 05:24:20.116775 1340 tablet.cc:2369] T 9b2018442bf0441f9bbe1c7e36dac068 P 4ef447bad3514908ab6c0b8e38159b5b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 05:24:20.158877 1585 raft_consensus.cc:482] T 9b2018442bf0441f9bbe1c7e36dac068 P 02ae7d5ca6e1473b8b0afccf84aca901 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 05:24:20.159016 1585 raft_consensus.cc:504] T 9b2018442bf0441f9bbe1c7e36dac068 P 02ae7d5ca6e1473b8b0afccf84aca901 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da01691963664f67b1c3a3b3724a935f" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 37423 } } peers { permanent_uuid: "02ae7d5ca6e1473b8b0afccf84aca901" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 35073 } } peers { permanent_uuid: "4ef447bad3514908ab6c0b8e38159b5b" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 33013 } }
I20260812 05:24:20.159431 1585 leader_election.cc:288] T 9b2018442bf0441f9bbe1c7e36dac068 P 02ae7d5ca6e1473b8b0afccf84aca901 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers da01691963664f67b1c3a3b3724a935f (127.31.100.3:37423), 4ef447bad3514908ab6c0b8e38159b5b (127.31.100.2:33013)
I20260812 05:24:20.162237 1577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:24:20.159693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.100.1:36171 (local address 127.31.100.2:33013)
0812 05:24:20.159742 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0812 05:24:20.159744 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 05:24:20.159752 (+ 8us) server_negotiation.cc:407] Connection header received
0812 05:24:20.159786 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:24:20.159787 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:24:20.159804 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:24:20.159872 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 05:24:20.160032 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:24:20.160509 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:24:20.161415 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:24:20.161937 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:24:20.161996 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:24:20.162005 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0812 05:24:20.162140 (+ 135us) server_negotiation.cc:299] Negotiation successful
0812 05:24:20.162178 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260812 05:24:20.162237 1119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:24:20.159615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.2:33013 (local address 127.31.100.1:36171)
0812 05:24:20.159660 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0812 05:24:20.159672 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0812 05:24:20.159716 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:24:20.159889 (+ 173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:24:20.159891 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:24:20.159893 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 05:24:20.159962 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:24:20.159964 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:24:20.160580 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:24:20.160581 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:24:20.161318 (+ 737us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:24:20.161320 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:24:20.162005 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:24:20.162007 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:24:20.162072 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:24:20.162075 (+ 3us) client_negotiation.cc:769] Sending connection context
0812 05:24:20.162119 (+ 44us) client_negotiation.cc:240] Negotiation successful
0812 05:24:20.162161 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25}
I20260812 05:24:20.162360 1576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:24:20.159689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.100.1:38575 (local address 127.31.100.3:37423)
0812 05:24:20.159734 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0812 05:24:20.159736 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 05:24:20.159915 (+ 179us) server_negotiation.cc:407] Connection header received
0812 05:24:20.159972 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:24:20.159973 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:24:20.159982 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:24:20.160030 (+ 48us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 05:24:20.160166 (+ 136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:24:20.160635 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:24:20.161579 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:24:20.162047 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:24:20.162110 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:24:20.162118 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0812 05:24:20.162257 (+ 139us) server_negotiation.cc:299] Negotiation successful
0812 05:24:20.162297 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260812 05:24:20.162366 1593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:24:20.159615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.3:37423 (local address 127.31.100.1:38575)
0812 05:24:20.159847 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0812 05:24:20.159854 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0812 05:24:20.159900 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:24:20.160040 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:24:20.160041 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:24:20.160042 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 05:24:20.160110 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:24:20.160112 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:24:20.160714 (+ 602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:24:20.160715 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:24:20.161486 (+ 771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:24:20.161488 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:24:20.162118 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:24:20.162119 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:24:20.162196 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:24:20.162199 (+ 3us) client_negotiation.cc:769] Sending connection context
0812 05:24:20.162245 (+ 46us) client_negotiation.cc:240] Negotiation successful
0812 05:24:20.162281 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":217,"mutex_wait_us":50,"thread_start_us":112,"threads_started":1}
I20260812 05:24:20.162755 1258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b2018442bf0441f9bbe1c7e36dac068" candidate_uuid: "02ae7d5ca6e1473b8b0afccf84aca901" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ef447bad3514908ab6c0b8e38159b5b" is_pre_election: true
I20260812 05:24:20.162803 1476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b2018442bf0441f9bbe1c7e36dac068" candidate_uuid: "02ae7d5ca6e1473b8b0afccf84aca901" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "da01691963664f67b1c3a3b3724a935f" is_pre_election: true
I20260812 05:24:20.162894 1258 raft_consensus.cc:2454] T 9b2018442bf0441f9bbe1c7e36dac068 P 4ef447bad3514908ab6c0b8e38159b5b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02ae7d5ca6e1473b8b0afccf84aca901 in term 0.
I20260812 05:24:20.162961 1476 raft_consensus.cc:2454] T 9b2018442bf0441f9bbe1c7e36dac068 P da01691963664f67b1c3a3b3724a935f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02ae7d5ca6e1473b8b0afccf84aca901 in term 0.
I20260812 05:24:20.163151 932 leader_election.cc:302] T 9b2018442bf0441f9bbe1c7e36dac068 P 02ae7d5ca6e1473b8b0afccf84aca901 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02ae7d5ca6e1473b8b0afccf84aca901, 4ef447bad3514908ab6c0b8e38159b5b; no voters:
I20260812 05:24:20.163285 1585 raft_consensus.cc:2781] T 9b2018442bf0441f9bbe1c7e36dac068 P 02ae7d5ca6e1473b8b0afccf84aca901 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 05:24:20.163357 1585 raft_consensus.cc:482] T 9b2018442bf0441f9bbe1c7e36dac068 P 02ae7d5ca6e1473b8b0afccf84aca901 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 05:24:20.163373 1585 raft_consensus.cc:3037] T 9b2018442bf0441f9bbe1c7e36dac068 P 02ae7d5ca6e1473b8b0afccf84aca901 [term 0 FOLLOWER]: Advancing to term 1
I20260812 05:24:20.164093 1585 raft_consensus.cc:504] T 9b2018442bf0441f9bbe1c7e36dac068 P 02ae7d5ca6e1473b8b0afccf84aca901 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da01691963664f67b1c3a3b3724a935f" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 37423 } } peers { permanent_uuid: "02ae7d5ca6e1473b8b0afccf84aca901" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 35073 } } peers { permanent_uuid: "4ef447bad3514908ab6c0b8e38159b5b" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 33013 } }
I20260812 05:24:20.164215 1585 leader_election.cc:288] T 9b2018442bf0441f9bbe1c7e36dac068 P 02ae7d5ca6e1473b8b0afccf84aca901 [CANDIDATE]: Term 1 election: Requested vote from peers da01691963664f67b1c3a3b3724a935f (127.31.100.3:37423), 4ef447bad3514908ab6c0b8e38159b5b (127.31.100.2:33013)
I20260812 05:24:20.164353 1476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b2018442bf0441f9bbe1c7e36dac068" candidate_uuid: "02ae7d5ca6e1473b8b0afccf84aca901" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "da01691963664f67b1c3a3b3724a935f"
I20260812 05:24:20.164372 1258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b2018442bf0441f9bbe1c7e36dac068" candidate_uuid: "02ae7d5ca6e1473b8b0afccf84aca901" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ef447bad3514908ab6c0b8e38159b5b"
I20260812 05:24:20.164434 1476 raft_consensus.cc:3037] T 9b2018442bf0441f9bbe1c7e36dac068 P da01691963664f67b1c3a3b3724a935f [term 0 FOLLOWER]: Advancing to term 1
I20260812 05:24:20.164444 1258 raft_consensus.cc:3037] T 9b2018442bf0441f9bbe1c7e36dac068 P 4ef447bad3514908ab6c0b8e38159b5b [term 0 FOLLOWER]: Advancing to term 1
I20260812 05:24:20.165326 1258 raft_consensus.cc:2454] T 9b2018442bf0441f9bbe1c7e36dac068 P 4ef447bad3514908ab6c0b8e38159b5b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02ae7d5ca6e1473b8b0afccf84aca901 in term 1.
I20260812 05:24:20.165321 1476 raft_consensus.cc:2454] T 9b2018442bf0441f9bbe1c7e36dac068 P da01691963664f67b1c3a3b3724a935f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02ae7d5ca6e1473b8b0afccf84aca901 in term 1.
I20260812 05:24:20.165521 932 leader_election.cc:302] T 9b2018442bf0441f9bbe1c7e36dac068 P 02ae7d5ca6e1473b8b0afccf84aca901 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 02ae7d5ca6e1473b8b0afccf84aca901, 4ef447bad3514908ab6c0b8e38159b5b, da01691963664f67b1c3a3b3724a935f; no voters:
I20260812 05:24:20.165614 1585 raft_consensus.cc:2781] T 9b2018442bf0441f9bbe1c7e36dac068 P 02ae7d5ca6e1473b8b0afccf84aca901 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 05:24:20.165731 1585 raft_consensus.cc:686] T 9b2018442bf0441f9bbe1c7e36dac068 P 02ae7d5ca6e1473b8b0afccf84aca901 [term 1 LEADER]: Becoming Leader. State: Replica: 02ae7d5ca6e1473b8b0afccf84aca901, State: Running, Role: LEADER
I20260812 05:24:20.165810 1585 consensus_queue.cc:237] T 9b2018442bf0441f9bbe1c7e36dac068 P 02ae7d5ca6e1473b8b0afccf84aca901 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da01691963664f67b1c3a3b3724a935f" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 37423 } } peers { permanent_uuid: "02ae7d5ca6e1473b8b0afccf84aca901" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 35073 } } peers { permanent_uuid: "4ef447bad3514908ab6c0b8e38159b5b" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 33013 } }
I20260812 05:24:20.166656 798 catalog_manager.cc:5495] T 9b2018442bf0441f9bbe1c7e36dac068 P 02ae7d5ca6e1473b8b0afccf84aca901 reported cstate change: term changed from 0 to 1, leader changed from <none> to 02ae7d5ca6e1473b8b0afccf84aca901 (127.31.100.1). New cstate: current_term: 1 leader_uuid: "02ae7d5ca6e1473b8b0afccf84aca901" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da01691963664f67b1c3a3b3724a935f" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 37423 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "02ae7d5ca6e1473b8b0afccf84aca901" member_type: VOTER last_known_addr { host: "127.31.100.1" port: 35073 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4ef447bad3514908ab6c0b8e38159b5b" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 33013 } health_report { overall_health: UNKNOWN } } }
I20260812 05:24:20.201903 1578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:24:20.199565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45026 (local address 127.31.100.1:35073)
0812 05:24:20.199631 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0812 05:24:20.199633 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 05:24:20.199644 (+ 11us) server_negotiation.cc:407] Connection header received
0812 05:24:20.199691 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:24:20.199692 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:24:20.199712 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:24:20.199772 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 05:24:20.199915 (+ 143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:24:20.200381 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:24:20.201113 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:24:20.201308 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:24:20.201364 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:24:20.201538 (+ 174us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 05:24:20.201649 (+ 111us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 05:24:20.201726 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0812 05:24:20.201778 (+ 52us) server_negotiation.cc:299] Negotiation successful
0812 05:24:20.201800 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260812 05:24:20.203795 1258 raft_consensus.cc:1261] T 9b2018442bf0441f9bbe1c7e36dac068 P 4ef447bad3514908ab6c0b8e38159b5b [term 1 FOLLOWER]: Refusing update from remote peer 02ae7d5ca6e1473b8b0afccf84aca901: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 05:24:20.203795 1476 raft_consensus.cc:1261] T 9b2018442bf0441f9bbe1c7e36dac068 P da01691963664f67b1c3a3b3724a935f [term 1 FOLLOWER]: Refusing update from remote peer 02ae7d5ca6e1473b8b0afccf84aca901: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 05:24:20.204186 1594 consensus_queue.cc:1035] T 9b2018442bf0441f9bbe1c7e36dac068 P 02ae7d5ca6e1473b8b0afccf84aca901 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4ef447bad3514908ab6c0b8e38159b5b" member_type: VOTER last_known_addr { host: "127.31.100.2" port: 33013 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 05:24:20.204331 1585 consensus_queue.cc:1035] T 9b2018442bf0441f9bbe1c7e36dac068 P 02ae7d5ca6e1473b8b0afccf84aca901 [LEADER]: Connected to new peer: Peer: permanent_uuid: "da01691963664f67b1c3a3b3724a935f" member_type: VOTER last_known_addr { host: "127.31.100.3" port: 37423 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 05:24:20.206908 1599 mvcc.cc:205] Tried to move back new op lower bound from 7317554217791651840 to 7317554217639743488. Current Snapshot: MvccSnapshot[applied={T|T < 7317554217791651840}]
I20260812 05:24:20.207012 1600 mvcc.cc:205] Tried to move back new op lower bound from 7317554217791651840 to 7317554217639743488. Current Snapshot: MvccSnapshot[applied={T|T < 7317554217791651840}]
I20260812 05:24:20.207244 1598 mvcc.cc:205] Tried to move back new op lower bound from 7317554217791651840 to 7317554217639743488. Current Snapshot: MvccSnapshot[applied={T|T < 7317554217791651840}]
I20260812 05:24:20.209475 796 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:53316:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 05:24:20.209560 796 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:53316:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 05:24:20.210287 796 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8813b7b835e04bb9ae74b0382c4f1ba6: Sending DeleteTablet for 3 replicas of tablet 9b2018442bf0441f9bbe1c7e36dac068
I20260812 05:24:20.210623 1450 tablet_service.cc:1514] Processing DeleteTablet for tablet 9b2018442bf0441f9bbe1c7e36dac068 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 05:24:20 UTC) from {username='slave'} at 127.0.0.1:53074
I20260812 05:24:20.210642 1223 tablet_service.cc:1514] Processing DeleteTablet for tablet 9b2018442bf0441f9bbe1c7e36dac068 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 05:24:20 UTC) from {username='slave'} at 127.0.0.1:46214
I20260812 05:24:20.210750 1013 tablet_service.cc:1514] Processing DeleteTablet for tablet 9b2018442bf0441f9bbe1c7e36dac068 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 05:24:20 UTC) from {username='slave'} at 127.0.0.1:45022
I20260812 05:24:20.210875 1623 tablet_replica.cc:330] T 9b2018442bf0441f9bbe1c7e36dac068 P da01691963664f67b1c3a3b3724a935f: stopping tablet replica
I20260812 05:24:20.210978 1625 tablet_replica.cc:330] T 9b2018442bf0441f9bbe1c7e36dac068 P 02ae7d5ca6e1473b8b0afccf84aca901: stopping tablet replica
I20260812 05:24:20.210979 1623 raft_consensus.cc:2229] T 9b2018442bf0441f9bbe1c7e36dac068 P da01691963664f67b1c3a3b3724a935f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 05:24:20.211071 1625 raft_consensus.cc:2229] T 9b2018442bf0441f9bbe1c7e36dac068 P 02ae7d5ca6e1473b8b0afccf84aca901 [term 1 LEADER]: Raft consensus shutting down.
I20260812 05:24:20.211078 1623 raft_consensus.cc:2258] T 9b2018442bf0441f9bbe1c7e36dac068 P da01691963664f67b1c3a3b3724a935f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 05:24:20.211175 1625 raft_consensus.cc:2258] T 9b2018442bf0441f9bbe1c7e36dac068 P 02ae7d5ca6e1473b8b0afccf84aca901 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 05:24:20.211380 32144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0waFvT/build/release/bin/kudu with pid 897
I20260812 05:24:20.222043 1623 ts_tablet_manager.cc:1905] T 9b2018442bf0441f9bbe1c7e36dac068 P da01691963664f67b1c3a3b3724a935f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 05:24:20.223515 1615 tablet_replica.cc:330] T 9b2018442bf0441f9bbe1c7e36dac068 P 4ef447bad3514908ab6c0b8e38159b5b: stopping tablet replica
I20260812 05:24:20.223579 1623 ts_tablet_manager.cc:1918] T 9b2018442bf0441f9bbe1c7e36dac068 P da01691963664f67b1c3a3b3724a935f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 05:24:20.223649 1615 raft_consensus.cc:2229] T 9b2018442bf0441f9bbe1c7e36dac068 P 4ef447bad3514908ab6c0b8e38159b5b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 05:24:20.223673 1623 log.cc:1196] T 9b2018442bf0441f9bbe1c7e36dac068 P da01691963664f67b1c3a3b3724a935f: Deleting WAL directory at /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786512017613366-32144-0/minicluster-data/ts-2/wal/wals/9b2018442bf0441f9bbe1c7e36dac068
I20260812 05:24:20.223768 1615 raft_consensus.cc:2258] T 9b2018442bf0441f9bbe1c7e36dac068 P 4ef447bad3514908ab6c0b8e38159b5b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 05:24:20.224031 1623 ts_tablet_manager.cc:1939] T 9b2018442bf0441f9bbe1c7e36dac068 P da01691963664f67b1c3a3b3724a935f: Deleting consensus metadata
I20260812 05:24:20.224326 1615 ts_tablet_manager.cc:1905] T 9b2018442bf0441f9bbe1c7e36dac068 P 4ef447bad3514908ab6c0b8e38159b5b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 05:24:20.224478 775 catalog_manager.cc:4841] TS da01691963664f67b1c3a3b3724a935f (127.31.100.3:37423): tablet 9b2018442bf0441f9bbe1c7e36dac068 (table test-table [id=59f462c5e64942c4986acb3f9c9868f3]) successfully deleted
I20260812 05:24:20.225513 1615 ts_tablet_manager.cc:1918] T 9b2018442bf0441f9bbe1c7e36dac068 P 4ef447bad3514908ab6c0b8e38159b5b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 05:24:20.225579 1615 log.cc:1196] T 9b2018442bf0441f9bbe1c7e36dac068 P 4ef447bad3514908ab6c0b8e38159b5b: Deleting WAL directory at /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786512017613366-32144-0/minicluster-data/ts-1/wal/wals/9b2018442bf0441f9bbe1c7e36dac068
I20260812 05:24:20.225788 1615 ts_tablet_manager.cc:1939] T 9b2018442bf0441f9bbe1c7e36dac068 P 4ef447bad3514908ab6c0b8e38159b5b: Deleting consensus metadata
I20260812 05:24:20.226156 778 catalog_manager.cc:4841] TS 4ef447bad3514908ab6c0b8e38159b5b (127.31.100.2:33013): tablet 9b2018442bf0441f9bbe1c7e36dac068 (table test-table [id=59f462c5e64942c4986acb3f9c9868f3]) successfully deleted
W20260812 05:24:20.226452 773 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.31.100.1:35073 (error 108)
I20260812 05:24:20.226917 1571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:24:20.226799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.1:35073 (local address 127.0.0.1:45032)
0812 05:24:20.226849 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0812 05:24:20.226888 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.100.1:35073: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260812 05:24:20.226958 32144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0waFvT/build/release/bin/kudu with pid 1120
W20260812 05:24:20.227046 773 catalog_manager.cc:4584] TS 02ae7d5ca6e1473b8b0afccf84aca901 (127.31.100.1:35073): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9b2018442bf0441f9bbe1c7e36dac068: Network error: Client connection negotiation failed: client connection to 127.31.100.1:35073: connect: Connection refused (error 111)
I20260812 05:24:20.234350 32144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0waFvT/build/release/bin/kudu with pid 1342
I20260812 05:24:20.249753 32144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0waFvT/build/release/bin/kudu with pid 754
2026-08-12T05:24:20Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (824 ms)
[----------] 6 tests from EncryptionPolicyTest (8757 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786512017613366-32144-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 05:24:20 dist-test-slave-42z9 krb5kdc[1633](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 05:24:20 dist-test-slave-42z9 krb5kdc[1633](info): set up 2 sockets
Aug 12 05:24:20 dist-test-slave-42z9 krb5kdc[1633](info): commencing operation
krb5kdc: starting...
W20260812 05:24:21.426951 32144 mini_kdc.cc:121] Time spent starting KDC: real 1.153s user 0.002s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786512017613366-32144-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786512017613366-32144-0/krb5kdc/test-user.keytab.
Aug 12 05:24:21 dist-test-slave-42z9 krb5kdc[1633](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512261, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T05:24:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T05:24:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786512017613366-32144-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786512017613366-32144-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260812 05:24:21.542054 32144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0waFvT/build/release/bin/kudu
/tmp/dist-test-task0waFvT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786512017613366-32144-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786512017613366-32144-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786512017613366-32144-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786512017613366-32144-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34221
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786512017613366-32144-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40899
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34221
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786512017613366-32144-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786512017613366-32144-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786512017613366-32144-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 05:24:21.621814 1657 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 05:24:21.622012 1657 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 05:24:21.622044 1657 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 05:24:21.623190 1657 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 05:24:21.623234 1657 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 05:24:21.623247 1657 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 05:24:21.623260 1657 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 05:24:21.623282 1657 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 05:24:21.624768 1657 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40899
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786512017613366-32144-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786512017613366-32144-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34221
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786512017613366-32144-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34221
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786512017613366-32144-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1657
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786512017613366-32144-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 04:20:34 UTC on ccycloud.quasar-phqdqp.root.comops.site
build id 4581871
I20260812 05:24:21.625000 1657 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 05:24:21.625216 1657 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 05:24:21.630021 1664 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 05:24:21.630015 1668 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 05:24:21.630044 1657 server_base.cc:1034] running on GCE node
W20260812 05:24:21.630015 1666 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 05:24:21.630411 1657 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 05:24:21.630716 1657 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 05:24:21.631855 1657 hybrid_clock.cc:648] HybridClock initialized: now 1786512261631835 us; error 36 us; skew 500 ppm
Aug 12 05:24:21 dist-test-slave-42z9 krb5kdc[1633](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512261, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 05:24:21.637718 1657 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260812 05:24:21.638186 1657 webserver.cc:460] Webserver started at http://127.0.0.1:35411/ using document root <none> and password file <none>
I20260812 05:24:21.638397 1657 fs_manager.cc:359] Metadata directory not provided
I20260812 05:24:21.638447 1657 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 05:24:21.638584 1657 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 05:24:21.639676 1657 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786512017613366-32144-0/minicluster-data/master-0/data/instance:
uuid: "9b719f463ad84110bd7bee765916edd3"
format_stamp: "Formatted at 2026-08-12 05:24:21 on dist-test-slave-42z9"
server_key: "b2286d549d700a1525dd5fd7643d0d32"
server_key_iv: "1cb2480d54b9b782d80d02f2a3c1be31"
server_key_version: "encryptionkey@0"
I20260812 05:24:21.640012 1657 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786512017613366-32144-0/minicluster-data/master-0/wal/instance:
uuid: "9b719f463ad84110bd7bee765916edd3"
format_stamp: "Formatted at 2026-08-12 05:24:21 on dist-test-slave-42z9"
server_key: "b2286d549d700a1525dd5fd7643d0d32"
server_key_iv: "1cb2480d54b9b782d80d02f2a3c1be31"
server_key_version: "encryptionkey@0"
I20260812 05:24:21.641268 1657 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260812 05:24:21.642021 1680 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 05:24:21.642175 1657 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260812 05:24:21.642263 1657 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786512017613366-32144-0/minicluster-data/master-0/data,/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786512017613366-32144-0/minicluster-data/master-0/wal
uuid: "9b719f463ad84110bd7bee765916edd3"
format_stamp: "Formatted at 2026-08-12 05:24:21 on dist-test-slave-42z9"
server_key: "b2286d549d700a1525dd5fd7643d0d32"
server_key_iv: "1cb2480d54b9b782d80d02f2a3c1be31"
server_key_version: "encryptionkey@0"
I20260812 05:24:21.642329 1657 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786512017613366-32144-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786512017613366-32144-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786512017613366-32144-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 05:24:21.658358 1657 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 05:24:21.660101 1657 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 05:24:21.660240 1657 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 05:24:21.664204 1657 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34221
I20260812 05:24:21.664238 1761 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34221 every 8 connection(s)
I20260812 05:24:21.664556 1657 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786512017613366-32144-0/minicluster-data/master-0/data/info.pb
I20260812 05:24:21.665413 1762 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 05:24:21.666110 32144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0waFvT/build/release/bin/kudu as pid 1657
I20260812 05:24:21.666217 32144 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786512017613366-32144-0/minicluster-data/master-0/wal/instance
I20260812 05:24:21.666412 32144 external_mini_cluster.cc:1428] Setting key 9802477eb75a203f0ff775fd4e172718
I20260812 05:24:21.668434 1762 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9b719f463ad84110bd7bee765916edd3: Bootstrap starting.
I20260812 05:24:21.669025 1762 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9b719f463ad84110bd7bee765916edd3: Neither blocks nor log segments found. Creating new log.
I20260812 05:24:21.669301 1762 log.cc:824] T 00000000000000000000000000000000 P 9b719f463ad84110bd7bee765916edd3: Log is configured to *not* fsync() on all Append() calls
I20260812 05:24:21.670065 1762 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9b719f463ad84110bd7bee765916edd3: No bootstrap required, opened a new log
I20260812 05:24:21.672356 1762 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9b719f463ad84110bd7bee765916edd3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b719f463ad84110bd7bee765916edd3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34221 } }
I20260812 05:24:21.672519 1762 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9b719f463ad84110bd7bee765916edd3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 05:24:21.672549 1762 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9b719f463ad84110bd7bee765916edd3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b719f463ad84110bd7bee765916edd3, State: Initialized, Role: FOLLOWER
I20260812 05:24:21.672672 1762 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9b719f463ad84110bd7bee765916edd3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b719f463ad84110bd7bee765916edd3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34221 } }
I20260812 05:24:21.672741 1762 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9b719f463ad84110bd7bee765916edd3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 05:24:21.672763 1762 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9b719f463ad84110bd7bee765916edd3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 05:24:21.672808 1762 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9b719f463ad84110bd7bee765916edd3 [term 0 FOLLOWER]: Advancing to term 1
Aug 12 05:24:21 dist-test-slave-42z9 krb5kdc[1633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512261, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260812 05:24:21.673353 1762 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9b719f463ad84110bd7bee765916edd3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b719f463ad84110bd7bee765916edd3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34221 } }
I20260812 05:24:21.673470 1762 leader_election.cc:302] T 00000000000000000000000000000000 P 9b719f463ad84110bd7bee765916edd3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9b719f463ad84110bd7bee765916edd3; no voters:
I20260812 05:24:21.673640 1762 leader_election.cc:288] T 00000000000000000000000000000000 P 9b719f463ad84110bd7bee765916edd3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 05:24:21.673671 1775 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9b719f463ad84110bd7bee765916edd3 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 05:24:21.673893 1762 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9b719f463ad84110bd7bee765916edd3 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 05:24:21.673959 1775 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9b719f463ad84110bd7bee765916edd3 [term 1 LEADER]: Becoming Leader. State: Replica: 9b719f463ad84110bd7bee765916edd3, State: Running, Role: LEADER
I20260812 05:24:21.674085 1775 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9b719f463ad84110bd7bee765916edd3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b719f463ad84110bd7bee765916edd3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34221 } }
I20260812 05:24:21.674445 1777 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b719f463ad84110bd7bee765916edd3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9b719f463ad84110bd7bee765916edd3. Latest consensus state: current_term: 1 leader_uuid: "9b719f463ad84110bd7bee765916edd3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b719f463ad84110bd7bee765916edd3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34221 } } }
I20260812 05:24:21.674443 1776 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b719f463ad84110bd7bee765916edd3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9b719f463ad84110bd7bee765916edd3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b719f463ad84110bd7bee765916edd3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34221 } } }
I20260812 05:24:21.674556 1777 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b719f463ad84110bd7bee765916edd3 [sys.catalog]: This master's current role is: LEADER
I20260812 05:24:21.674579 1776 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b719f463ad84110bd7bee765916edd3 [sys.catalog]: This master's current role is: LEADER
W20260812 05:24:21.675879 1803 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9b719f463ad84110bd7bee765916edd3: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 05:24:21.675935 1803 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 05:24:21.675998 1791 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 05:24:21.676156 1791 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 05:24:21.677098 1770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:24:21.666938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54783 (local address 127.0.0.1:34221)
0812 05:24:21.667096 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0812 05:24:21.667110 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0812 05:24:21.667135 (+ 25us) server_negotiation.cc:407] Connection header received
0812 05:24:21.667541 (+ 406us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:24:21.667547 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:24:21.667593 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:24:21.667708 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:24:21.668469 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:24:21.669411 (+ 942us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:24:21.670419 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:24:21.670823 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:24:21.670899 (+ 76us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:24:21.673579 (+ 2680us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 05:24:21.673584 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 05:24:21.673584 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 05:24:21.673597 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0812 05:24:21.675565 (+ 1968us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:24:21.675949 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:24:21.675952 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:24:21.675952 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:24:21.676062 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:24:21.676268 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:24:21.676269 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:24:21.676270 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:24:21.676497 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 05:24:21.676629 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0812 05:24:21.676791 (+ 162us) server_negotiation.cc:299] Negotiation successful
0812 05:24:21.676897 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":65,"threads_started":1}
I20260812 05:24:21.677840 1791 catalog_manager.cc:1347] Generated new cluster ID: 4d2c8f0c17b84e959f6845bf73d6d090
I20260812 05:24:21.677891 1791 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 05:24:21.678270 32144 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 05:24:21.687204 1770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:24:21.681749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34789 (local address 127.0.0.1:34221)
0812 05:24:21.681787 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0812 05:24:21.681789 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 05:24:21.681910 (+ 121us) server_negotiation.cc:407] Connection header received
0812 05:24:21.681952 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:24:21.681953 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:24:21.681962 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:24:21.682006 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:24:21.682467 (+ 461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:24:21.682945 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:24:21.683594 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:24:21.683823 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:24:21.683878 (+ 55us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:24:21.684923 (+ 1045us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 05:24:21.684925 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 05:24:21.684925 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 05:24:21.684932 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 05:24:21.686163 (+ 1231us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:24:21.686489 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:24:21.686490 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:24:21.686490 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:24:21.686543 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:24:21.686736 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:24:21.686737 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:24:21.686737 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:24:21.686838 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 05:24:21.686923 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0812 05:24:21.687055 (+ 132us) server_negotiation.cc:299] Negotiation successful
0812 05:24:21.687123 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260812 05:24:21.692852 1791 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 05:24:21.693389 1791 catalog_manager.cc:1504] Loading token signing keys...
I20260812 05:24:21.697628 1791 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9b719f463ad84110bd7bee765916edd3: Generated new TSK 0
I20260812 05:24:21.697793 1791 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 05:24:21.701588 32144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0waFvT/build/release/bin/kudu with pid 1657
W20260812 05:24:21.727010 1630 connection.cc:535] client connection to 127.0.0.1:34221 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:34221): Connection reset by peer (error 104)
2026-08-12T05:24:21Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1466 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786512017613366-32144-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 05:24:21 dist-test-slave-42z9 krb5kdc[1822](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 05:24:21 dist-test-slave-42z9 krb5kdc[1822](info): set up 2 sockets
Aug 12 05:24:21 dist-test-slave-42z9 krb5kdc[1822](info): commencing operation
krb5kdc: starting...
W20260812 05:24:22.883597 32144 mini_kdc.cc:121] Time spent starting KDC: real 1.143s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786512017613366-32144-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786512017613366-32144-0/krb5kdc/test-user.keytab.
Aug 12 05:24:22 dist-test-slave-42z9 krb5kdc[1822](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512262, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T05:24:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T05:24:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786512017613366-32144-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786512017613366-32144-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260812 05:24:22.997386 32144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0waFvT/build/release/bin/kudu
/tmp/dist-test-task0waFvT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786512017613366-32144-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786512017613366-32144-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786512017613366-32144-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786512017613366-32144-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41377
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786512017613366-32144-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:32931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41377
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786512017613366-32144-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786512017613366-32144-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786512017613366-32144-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 05:24:23.078285 1848 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 05:24:23.078488 1848 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 05:24:23.078521 1848 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 05:24:23.079649 1848 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 05:24:23.079691 1848 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 05:24:23.079710 1848 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 05:24:23.079725 1848 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 05:24:23.079741 1848 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 05:24:23.081228 1848 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:32931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786512017613366-32144-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786512017613366-32144-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41377
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786512017613366-32144-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41377
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786512017613366-32144-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1848
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786512017613366-32144-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 04:20:34 UTC on ccycloud.quasar-phqdqp.root.comops.site
build id 4581871
I20260812 05:24:23.081496 1848 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 05:24:23.081713 1848 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 05:24:23.086455 1861 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 05:24:23.086513 1848 server_base.cc:1034] running on GCE node
W20260812 05:24:23.086519 1859 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 05:24:23.086532 1856 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 05:24:23.086894 1848 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 05:24:23.087195 1848 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 05:24:23.088340 1848 hybrid_clock.cc:648] HybridClock initialized: now 1786512263088326 us; error 30 us; skew 500 ppm
Aug 12 05:24:23 dist-test-slave-42z9 krb5kdc[1822](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512263, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 05:24:23.090827 1848 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260812 05:24:23.091226 1848 webserver.cc:460] Webserver started at http://127.0.0.1:35421/ using document root <none> and password file <none>
I20260812 05:24:23.091437 1848 fs_manager.cc:359] Metadata directory not provided
I20260812 05:24:23.091482 1848 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 05:24:23.091599 1848 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 05:24:23.092612 1848 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786512017613366-32144-0/minicluster-data/master-0/data/instance:
uuid: "7445966cc5be46a4baa50e5ac23f8420"
format_stamp: "Formatted at 2026-08-12 05:24:23 on dist-test-slave-42z9"
server_key: "823682233c50761bc053fa7ca63247c5"
server_key_iv: "22c32f5264a310a1fa10662da164351d"
server_key_version: "encryptionkey@0"
I20260812 05:24:23.092922 1848 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786512017613366-32144-0/minicluster-data/master-0/wal/instance:
uuid: "7445966cc5be46a4baa50e5ac23f8420"
format_stamp: "Formatted at 2026-08-12 05:24:23 on dist-test-slave-42z9"
server_key: "823682233c50761bc053fa7ca63247c5"
server_key_iv: "22c32f5264a310a1fa10662da164351d"
server_key_version: "encryptionkey@0"
I20260812 05:24:23.094202 1848 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260812 05:24:23.094856 1874 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 05:24:23.094997 1848 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260812 05:24:23.095083 1848 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786512017613366-32144-0/minicluster-data/master-0/data,/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786512017613366-32144-0/minicluster-data/master-0/wal
uuid: "7445966cc5be46a4baa50e5ac23f8420"
format_stamp: "Formatted at 2026-08-12 05:24:23 on dist-test-slave-42z9"
server_key: "823682233c50761bc053fa7ca63247c5"
server_key_iv: "22c32f5264a310a1fa10662da164351d"
server_key_version: "encryptionkey@0"
I20260812 05:24:23.095144 1848 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786512017613366-32144-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786512017613366-32144-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786512017613366-32144-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 05:24:23.106895 1848 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 05:24:23.108897 1848 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 05:24:23.109028 1848 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 05:24:23.112982 1848 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41377
I20260812 05:24:23.113029 1971 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41377 every 8 connection(s)
I20260812 05:24:23.113328 1848 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786512017613366-32144-0/minicluster-data/master-0/data/info.pb
I20260812 05:24:23.114228 1973 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 05:24:23.116627 1973 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7445966cc5be46a4baa50e5ac23f8420: Bootstrap starting.
I20260812 05:24:23.117209 1973 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7445966cc5be46a4baa50e5ac23f8420: Neither blocks nor log segments found. Creating new log.
I20260812 05:24:23.117543 1973 log.cc:824] T 00000000000000000000000000000000 P 7445966cc5be46a4baa50e5ac23f8420: Log is configured to *not* fsync() on all Append() calls
I20260812 05:24:23.118196 1973 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7445966cc5be46a4baa50e5ac23f8420: No bootstrap required, opened a new log
I20260812 05:24:23.119822 1973 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7445966cc5be46a4baa50e5ac23f8420 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7445966cc5be46a4baa50e5ac23f8420" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41377 } }
I20260812 05:24:23.119956 1973 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7445966cc5be46a4baa50e5ac23f8420 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 05:24:23.119989 1973 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7445966cc5be46a4baa50e5ac23f8420 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7445966cc5be46a4baa50e5ac23f8420, State: Initialized, Role: FOLLOWER
I20260812 05:24:23.120128 1973 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7445966cc5be46a4baa50e5ac23f8420 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7445966cc5be46a4baa50e5ac23f8420" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41377 } }
I20260812 05:24:23.120190 1973 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7445966cc5be46a4baa50e5ac23f8420 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 05:24:23.120219 1973 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7445966cc5be46a4baa50e5ac23f8420 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 05:24:23.120266 1973 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7445966cc5be46a4baa50e5ac23f8420 [term 0 FOLLOWER]: Advancing to term 1
I20260812 05:24:23.120786 1973 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7445966cc5be46a4baa50e5ac23f8420 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7445966cc5be46a4baa50e5ac23f8420" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41377 } }
I20260812 05:24:23.120878 1973 leader_election.cc:302] T 00000000000000000000000000000000 P 7445966cc5be46a4baa50e5ac23f8420 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7445966cc5be46a4baa50e5ac23f8420; no voters:
I20260812 05:24:23.121029 1973 leader_election.cc:288] T 00000000000000000000000000000000 P 7445966cc5be46a4baa50e5ac23f8420 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 05:24:23.121063 1978 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7445966cc5be46a4baa50e5ac23f8420 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 05:24:23.121191 1978 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7445966cc5be46a4baa50e5ac23f8420 [term 1 LEADER]: Becoming Leader. State: Replica: 7445966cc5be46a4baa50e5ac23f8420, State: Running, Role: LEADER
I20260812 05:24:23.121272 1973 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7445966cc5be46a4baa50e5ac23f8420 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 05:24:23.121282 1978 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7445966cc5be46a4baa50e5ac23f8420 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7445966cc5be46a4baa50e5ac23f8420" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41377 } }
I20260812 05:24:23.121733 1981 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7445966cc5be46a4baa50e5ac23f8420 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7445966cc5be46a4baa50e5ac23f8420. Latest consensus state: current_term: 1 leader_uuid: "7445966cc5be46a4baa50e5ac23f8420" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7445966cc5be46a4baa50e5ac23f8420" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41377 } } }
I20260812 05:24:23.121852 1980 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7445966cc5be46a4baa50e5ac23f8420 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7445966cc5be46a4baa50e5ac23f8420" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7445966cc5be46a4baa50e5ac23f8420" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41377 } } }
I20260812 05:24:23.121874 1981 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7445966cc5be46a4baa50e5ac23f8420 [sys.catalog]: This master's current role is: LEADER
I20260812 05:24:23.121942 32144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0waFvT/build/release/bin/kudu as pid 1848
I20260812 05:24:23.122071 32144 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786512017613366-32144-0/minicluster-data/master-0/wal/instance
I20260812 05:24:23.122139 1991 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 05:24:23.122145 1980 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7445966cc5be46a4baa50e5ac23f8420 [sys.catalog]: This master's current role is: LEADER
I20260812 05:24:23.122270 32144 external_mini_cluster.cc:1428] Setting key a81ca809167a5c31ea79d0568c186def
I20260812 05:24:23.122694 1991 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 05:24:23.124226 1991 catalog_manager.cc:1347] Generated new cluster ID: cd02ddc95db44d4287940adb4271be0b
I20260812 05:24:23.124272 1991 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 05:24:23.128273 2004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:24:23.122723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42045 (local address 127.0.0.1:41377)
0812 05:24:23.123047 (+ 324us) server_negotiation.cc:206] Beginning negotiation
0812 05:24:23.123056 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 05:24:23.123092 (+ 36us) server_negotiation.cc:407] Connection header received
0812 05:24:23.123365 (+ 273us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:24:23.123375 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:24:23.123442 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:24:23.123583 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:24:23.123778 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:24:23.125127 (+ 1349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:24:23.126222 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:24:23.126864 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:24:23.126989 (+ 125us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:24:23.127620 (+ 631us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 05:24:23.127624 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 05:24:23.127625 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 05:24:23.127826 (+ 201us) server_negotiation.cc:922] calling sasl_server_start()
0812 05:24:23.127864 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 05:24:23.127878 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 05:24:23.127963 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0812 05:24:23.128035 (+ 72us) server_negotiation.cc:299] Negotiation successful
0812 05:24:23.128064 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":218,"thread_start_us":67,"threads_started":1}
I20260812 05:24:23.129238 32144 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 05:24:23.133989 2004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:24:23.130788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57317 (local address 127.0.0.1:41377)
0812 05:24:23.130825 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0812 05:24:23.130827 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 05:24:23.130858 (+ 31us) server_negotiation.cc:407] Connection header received
0812 05:24:23.130915 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:24:23.130916 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:24:23.130925 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:24:23.130982 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:24:23.131095 (+ 113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:24:23.131810 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:24:23.132766 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:24:23.133061 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:24:23.133124 (+ 63us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:24:23.133767 (+ 643us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 05:24:23.133769 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 05:24:23.133769 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 05:24:23.133773 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0812 05:24:23.133788 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 05:24:23.133794 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 05:24:23.133853 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0812 05:24:23.133904 (+ 51us) server_negotiation.cc:299] Negotiation successful
0812 05:24:23.133916 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260812 05:24:23.150202 1991 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 05:24:23.150882 1991 catalog_manager.cc:1504] Loading token signing keys...
I20260812 05:24:23.164640 1991 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7445966cc5be46a4baa50e5ac23f8420: Generated new TSK 0
I20260812 05:24:23.164832 1991 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 05:24:23.171721 32144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0waFvT/build/release/bin/kudu with pid 1848
W20260812 05:24:23.185091 1819 connection.cc:535] client connection to 127.0.0.1:41377 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:41377): Connection reset by peer (error 104)
2026-08-12T05:24:23Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1458 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786512017613366-32144-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 05:24:23 dist-test-slave-42z9 krb5kdc[2022](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 05:24:23 dist-test-slave-42z9 krb5kdc[2022](info): set up 2 sockets
Aug 12 05:24:23 dist-test-slave-42z9 krb5kdc[2022](info): commencing operation
krb5kdc: starting...
W20260812 05:24:24.331300 32144 mini_kdc.cc:121] Time spent starting KDC: real 1.133s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786512017613366-32144-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786512017613366-32144-0/krb5kdc/test-user.keytab.
Aug 12 05:24:24 dist-test-slave-42z9 krb5kdc[2022](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512264, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T05:24:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T05:24:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786512017613366-32144-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786512017613366-32144-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260812 05:24:24.445761 32144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0waFvT/build/release/bin/kudu
/tmp/dist-test-task0waFvT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786512017613366-32144-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786512017613366-32144-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786512017613366-32144-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786512017613366-32144-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45575
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786512017613366-32144-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38453
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45575
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786512017613366-32144-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786512017613366-32144-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786512017613366-32144-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 05:24:24.524840 2041 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 05:24:24.525036 2041 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 05:24:24.525058 2041 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 05:24:24.526171 2041 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 05:24:24.526211 2041 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 05:24:24.526222 2041 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 05:24:24.526233 2041 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 05:24:24.526247 2041 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 05:24:24.527626 2041 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38453
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786512017613366-32144-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786512017613366-32144-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45575
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786512017613366-32144-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45575
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786512017613366-32144-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2041
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786512017613366-32144-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 04:20:34 UTC on ccycloud.quasar-phqdqp.root.comops.site
build id 4581871
I20260812 05:24:24.527808 2041 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 05:24:24.527968 2041 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 05:24:24.532598 2054 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 05:24:24.532603 2053 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 05:24:24.532603 2056 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 05:24:24.532684 2041 server_base.cc:1034] running on GCE node
I20260812 05:24:24.533041 2041 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 05:24:24.533290 2041 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 05:24:24.534418 2041 hybrid_clock.cc:648] HybridClock initialized: now 1786512264534406 us; error 28 us; skew 500 ppm
Aug 12 05:24:24 dist-test-slave-42z9 krb5kdc[2022](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512264, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 05:24:24.536875 2041 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260812 05:24:24.537263 2041 webserver.cc:460] Webserver started at http://127.0.0.1:39621/ using document root <none> and password file <none>
I20260812 05:24:24.537494 2041 fs_manager.cc:359] Metadata directory not provided
I20260812 05:24:24.537545 2041 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 05:24:24.537678 2041 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 05:24:24.538651 2041 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786512017613366-32144-0/minicluster-data/master-0/data/instance:
uuid: "12ee989d84824d87a5ce0b22ad7a4ed6"
format_stamp: "Formatted at 2026-08-12 05:24:24 on dist-test-slave-42z9"
server_key: "ca1f1f575567298bd2d2e08ba6165761"
server_key_iv: "02c5b6055769a3991cfa8866a4042527"
server_key_version: "encryptionkey@0"
I20260812 05:24:24.538973 2041 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786512017613366-32144-0/minicluster-data/master-0/wal/instance:
uuid: "12ee989d84824d87a5ce0b22ad7a4ed6"
format_stamp: "Formatted at 2026-08-12 05:24:24 on dist-test-slave-42z9"
server_key: "ca1f1f575567298bd2d2e08ba6165761"
server_key_iv: "02c5b6055769a3991cfa8866a4042527"
server_key_version: "encryptionkey@0"
I20260812 05:24:24.540261 2041 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260812 05:24:24.540925 2072 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 05:24:24.541064 2041 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260812 05:24:24.541147 2041 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786512017613366-32144-0/minicluster-data/master-0/data,/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786512017613366-32144-0/minicluster-data/master-0/wal
uuid: "12ee989d84824d87a5ce0b22ad7a4ed6"
format_stamp: "Formatted at 2026-08-12 05:24:24 on dist-test-slave-42z9"
server_key: "ca1f1f575567298bd2d2e08ba6165761"
server_key_iv: "02c5b6055769a3991cfa8866a4042527"
server_key_version: "encryptionkey@0"
I20260812 05:24:24.541208 2041 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786512017613366-32144-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786512017613366-32144-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786512017613366-32144-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 05:24:24.544344 2041 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 05:24:24.546377 2041 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 05:24:24.546506 2041 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 05:24:24.550199 2041 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45575
I20260812 05:24:24.550228 2149 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45575 every 8 connection(s)
I20260812 05:24:24.550505 2041 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786512017613366-32144-0/minicluster-data/master-0/data/info.pb
I20260812 05:24:24.551338 2150 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 05:24:24.553514 2150 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 12ee989d84824d87a5ce0b22ad7a4ed6: Bootstrap starting.
I20260812 05:24:24.554091 2150 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 12ee989d84824d87a5ce0b22ad7a4ed6: Neither blocks nor log segments found. Creating new log.
I20260812 05:24:24.554363 2150 log.cc:824] T 00000000000000000000000000000000 P 12ee989d84824d87a5ce0b22ad7a4ed6: Log is configured to *not* fsync() on all Append() calls
I20260812 05:24:24.554980 2150 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 12ee989d84824d87a5ce0b22ad7a4ed6: No bootstrap required, opened a new log
I20260812 05:24:24.556511 2150 raft_consensus.cc:348] T 00000000000000000000000000000000 P 12ee989d84824d87a5ce0b22ad7a4ed6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12ee989d84824d87a5ce0b22ad7a4ed6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45575 } }
I20260812 05:24:24.556665 2150 raft_consensus.cc:374] T 00000000000000000000000000000000 P 12ee989d84824d87a5ce0b22ad7a4ed6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 05:24:24.556710 2150 raft_consensus.cc:729] T 00000000000000000000000000000000 P 12ee989d84824d87a5ce0b22ad7a4ed6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12ee989d84824d87a5ce0b22ad7a4ed6, State: Initialized, Role: FOLLOWER
I20260812 05:24:24.556819 2150 consensus_queue.cc:260] T 00000000000000000000000000000000 P 12ee989d84824d87a5ce0b22ad7a4ed6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12ee989d84824d87a5ce0b22ad7a4ed6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45575 } }
I20260812 05:24:24.556893 2150 raft_consensus.cc:388] T 00000000000000000000000000000000 P 12ee989d84824d87a5ce0b22ad7a4ed6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 05:24:24.556919 2150 raft_consensus.cc:482] T 00000000000000000000000000000000 P 12ee989d84824d87a5ce0b22ad7a4ed6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 05:24:24.556952 2150 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 12ee989d84824d87a5ce0b22ad7a4ed6 [term 0 FOLLOWER]: Advancing to term 1
I20260812 05:24:24.557405 2150 raft_consensus.cc:504] T 00000000000000000000000000000000 P 12ee989d84824d87a5ce0b22ad7a4ed6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12ee989d84824d87a5ce0b22ad7a4ed6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45575 } }
I20260812 05:24:24.557526 2150 leader_election.cc:302] T 00000000000000000000000000000000 P 12ee989d84824d87a5ce0b22ad7a4ed6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 12ee989d84824d87a5ce0b22ad7a4ed6; no voters:
I20260812 05:24:24.557658 2150 leader_election.cc:288] T 00000000000000000000000000000000 P 12ee989d84824d87a5ce0b22ad7a4ed6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 05:24:24.557716 2153 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 12ee989d84824d87a5ce0b22ad7a4ed6 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 05:24:24.557868 2150 sys_catalog.cc:564] T 00000000000000000000000000000000 P 12ee989d84824d87a5ce0b22ad7a4ed6 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 05:24:24.558064 2153 raft_consensus.cc:686] T 00000000000000000000000000000000 P 12ee989d84824d87a5ce0b22ad7a4ed6 [term 1 LEADER]: Becoming Leader. State: Replica: 12ee989d84824d87a5ce0b22ad7a4ed6, State: Running, Role: LEADER
I20260812 05:24:24.558171 2153 consensus_queue.cc:237] T 00000000000000000000000000000000 P 12ee989d84824d87a5ce0b22ad7a4ed6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12ee989d84824d87a5ce0b22ad7a4ed6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45575 } }
I20260812 05:24:24.558516 2154 sys_catalog.cc:455] T 00000000000000000000000000000000 P 12ee989d84824d87a5ce0b22ad7a4ed6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "12ee989d84824d87a5ce0b22ad7a4ed6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12ee989d84824d87a5ce0b22ad7a4ed6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45575 } } }
I20260812 05:24:24.558545 2156 sys_catalog.cc:455] T 00000000000000000000000000000000 P 12ee989d84824d87a5ce0b22ad7a4ed6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 12ee989d84824d87a5ce0b22ad7a4ed6. Latest consensus state: current_term: 1 leader_uuid: "12ee989d84824d87a5ce0b22ad7a4ed6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12ee989d84824d87a5ce0b22ad7a4ed6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45575 } } }
I20260812 05:24:24.558595 2154 sys_catalog.cc:458] T 00000000000000000000000000000000 P 12ee989d84824d87a5ce0b22ad7a4ed6 [sys.catalog]: This master's current role is: LEADER
I20260812 05:24:24.558646 2156 sys_catalog.cc:458] T 00000000000000000000000000000000 P 12ee989d84824d87a5ce0b22ad7a4ed6 [sys.catalog]: This master's current role is: LEADER
I20260812 05:24:24.558830 2169 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 05:24:24.559525 2169 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 05:24:24.559895 32144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0waFvT/build/release/bin/kudu as pid 2041
I20260812 05:24:24.560002 32144 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786512017613366-32144-0/minicluster-data/master-0/wal/instance
I20260812 05:24:24.560177 32144 external_mini_cluster.cc:1428] Setting key e035357d7f4d03a1f8f8caa18c3c7d4b
I20260812 05:24:24.561936 2169 catalog_manager.cc:1347] Generated new cluster ID: b8bf06927c1c4d2d921090b399d54f01
I20260812 05:24:24.561982 2169 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 05:24:24.562584 2189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:24:24.560575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57975 (local address 127.0.0.1:45575)
0812 05:24:24.560901 (+ 326us) server_negotiation.cc:206] Beginning negotiation
0812 05:24:24.560910 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 05:24:24.560943 (+ 33us) server_negotiation.cc:407] Connection header received
0812 05:24:24.561050 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:24:24.561057 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:24:24.561085 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:24:24.561235 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:24:24.562069 (+ 834us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 05:24:24.562072 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 05:24:24.562073 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 05:24:24.562077 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0812 05:24:24.562117 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 05:24:24.562132 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 05:24:24.562185 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0812 05:24:24.562354 (+ 169us) server_negotiation.cc:299] Negotiation successful
0812 05:24:24.562395 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":278,"thread_start_us":89,"threads_started":1}
I20260812 05:24:24.563544 32144 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 05:24:24.566035 2189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:24:24.565035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41287 (local address 127.0.0.1:45575)
0812 05:24:24.565067 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0812 05:24:24.565069 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 05:24:24.565173 (+ 104us) server_negotiation.cc:407] Connection header received
0812 05:24:24.565215 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:24:24.565216 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:24:24.565221 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:24:24.565274 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:24:24.565834 (+ 560us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 05:24:24.565836 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 05:24:24.565836 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 05:24:24.565840 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0812 05:24:24.565857 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 05:24:24.565861 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 05:24:24.565917 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0812 05:24:24.565964 (+ 47us) server_negotiation.cc:299] Negotiation successful
0812 05:24:24.565976 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260812 05:24:24.584916 2169 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 05:24:24.585085 2169 catalog_manager.cc:1504] Loading token signing keys...
I20260812 05:24:24.588354 2169 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 12ee989d84824d87a5ce0b22ad7a4ed6: Generated new TSK 0
I20260812 05:24:24.588512 2169 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 05:24:24.592330 32144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0waFvT/build/release/bin/kudu with pid 2041
2026-08-12T05:24:24Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1420 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786512017613366-32144-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 05:24:24 dist-test-slave-42z9 krb5kdc[2202](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 05:24:24 dist-test-slave-42z9 krb5kdc[2202](info): set up 2 sockets
Aug 12 05:24:24 dist-test-slave-42z9 krb5kdc[2202](info): commencing operation
krb5kdc: starting...
W20260812 05:24:25.760164 32144 mini_kdc.cc:121] Time spent starting KDC: real 1.141s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786512017613366-32144-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786512017613366-32144-0/krb5kdc/test-user.keytab.
Aug 12 05:24:25 dist-test-slave-42z9 krb5kdc[2202](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512265, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T05:24:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T05:24:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786512017613366-32144-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786512017613366-32144-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260812 05:24:25.871572 32144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0waFvT/build/release/bin/kudu
/tmp/dist-test-task0waFvT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786512017613366-32144-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786512017613366-32144-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786512017613366-32144-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786512017613366-32144-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40543
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786512017613366-32144-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38029
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40543
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786512017613366-32144-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786512017613366-32144-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786512017613366-32144-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 05:24:25.951709 2221 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 05:24:25.951908 2221 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 05:24:25.951929 2221 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 05:24:25.952996 2221 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 05:24:25.953030 2221 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 05:24:25.953042 2221 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 05:24:25.953053 2221 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 05:24:25.953069 2221 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 05:24:25.954504 2221 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38029
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786512017613366-32144-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786512017613366-32144-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40543
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786512017613366-32144-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40543
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786512017613366-32144-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2221
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786512017613366-32144-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 04:20:34 UTC on ccycloud.quasar-phqdqp.root.comops.site
build id 4581871
I20260812 05:24:25.954689 2221 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 05:24:25.954852 2221 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 05:24:25.959374 2233 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 05:24:25.959422 2229 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 05:24:25.959381 2227 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 05:24:25.959434 2221 server_base.cc:1034] running on GCE node
I20260812 05:24:25.959857 2221 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 05:24:25.960124 2221 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 05:24:25.961262 2221 hybrid_clock.cc:648] HybridClock initialized: now 1786512265961251 us; error 30 us; skew 500 ppm
Aug 12 05:24:25 dist-test-slave-42z9 krb5kdc[2202](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512265, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 05:24:25.963557 2221 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260812 05:24:25.963965 2221 webserver.cc:460] Webserver started at http://127.0.0.1:39313/ using document root <none> and password file <none>
I20260812 05:24:25.964133 2221 fs_manager.cc:359] Metadata directory not provided
I20260812 05:24:25.964169 2221 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 05:24:25.964257 2221 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 05:24:25.965194 2221 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786512017613366-32144-0/minicluster-data/master-0/data/instance:
uuid: "33cb5e2d82544e70adcb8dd2dfbbfb0e"
format_stamp: "Formatted at 2026-08-12 05:24:25 on dist-test-slave-42z9"
server_key: "d929c637cf856a242516d282bc8869a2"
server_key_iv: "965d6b71f0e5d87da2493512d88c82c0"
server_key_version: "encryptionkey@0"
I20260812 05:24:25.965524 2221 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786512017613366-32144-0/minicluster-data/master-0/wal/instance:
uuid: "33cb5e2d82544e70adcb8dd2dfbbfb0e"
format_stamp: "Formatted at 2026-08-12 05:24:25 on dist-test-slave-42z9"
server_key: "d929c637cf856a242516d282bc8869a2"
server_key_iv: "965d6b71f0e5d87da2493512d88c82c0"
server_key_version: "encryptionkey@0"
I20260812 05:24:25.966684 2221 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260812 05:24:25.967298 2243 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 05:24:25.967439 2221 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260812 05:24:25.967525 2221 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786512017613366-32144-0/minicluster-data/master-0/data,/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786512017613366-32144-0/minicluster-data/master-0/wal
uuid: "33cb5e2d82544e70adcb8dd2dfbbfb0e"
format_stamp: "Formatted at 2026-08-12 05:24:25 on dist-test-slave-42z9"
server_key: "d929c637cf856a242516d282bc8869a2"
server_key_iv: "965d6b71f0e5d87da2493512d88c82c0"
server_key_version: "encryptionkey@0"
I20260812 05:24:25.967592 2221 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786512017613366-32144-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786512017613366-32144-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786512017613366-32144-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 05:24:26.000468 2221 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 05:24:26.002669 2221 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 05:24:26.002784 2221 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 05:24:26.006600 2221 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40543
I20260812 05:24:26.006659 2327 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40543 every 8 connection(s)
I20260812 05:24:26.006971 2221 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786512017613366-32144-0/minicluster-data/master-0/data/info.pb
I20260812 05:24:26.007824 2328 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 05:24:26.010203 2328 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 33cb5e2d82544e70adcb8dd2dfbbfb0e: Bootstrap starting.
I20260812 05:24:26.010789 2328 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 33cb5e2d82544e70adcb8dd2dfbbfb0e: Neither blocks nor log segments found. Creating new log.
I20260812 05:24:26.011061 2328 log.cc:824] T 00000000000000000000000000000000 P 33cb5e2d82544e70adcb8dd2dfbbfb0e: Log is configured to *not* fsync() on all Append() calls
I20260812 05:24:26.011778 2328 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 33cb5e2d82544e70adcb8dd2dfbbfb0e: No bootstrap required, opened a new log
I20260812 05:24:26.013377 2328 raft_consensus.cc:348] T 00000000000000000000000000000000 P 33cb5e2d82544e70adcb8dd2dfbbfb0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33cb5e2d82544e70adcb8dd2dfbbfb0e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40543 } }
I20260812 05:24:26.013552 2328 raft_consensus.cc:374] T 00000000000000000000000000000000 P 33cb5e2d82544e70adcb8dd2dfbbfb0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 05:24:26.013589 2328 raft_consensus.cc:729] T 00000000000000000000000000000000 P 33cb5e2d82544e70adcb8dd2dfbbfb0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33cb5e2d82544e70adcb8dd2dfbbfb0e, State: Initialized, Role: FOLLOWER
I20260812 05:24:26.013733 2328 consensus_queue.cc:260] T 00000000000000000000000000000000 P 33cb5e2d82544e70adcb8dd2dfbbfb0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33cb5e2d82544e70adcb8dd2dfbbfb0e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40543 } }
I20260812 05:24:26.013813 2328 raft_consensus.cc:388] T 00000000000000000000000000000000 P 33cb5e2d82544e70adcb8dd2dfbbfb0e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 05:24:26.013850 2328 raft_consensus.cc:482] T 00000000000000000000000000000000 P 33cb5e2d82544e70adcb8dd2dfbbfb0e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 05:24:26.013898 2328 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 33cb5e2d82544e70adcb8dd2dfbbfb0e [term 0 FOLLOWER]: Advancing to term 1
I20260812 05:24:26.014432 2328 raft_consensus.cc:504] T 00000000000000000000000000000000 P 33cb5e2d82544e70adcb8dd2dfbbfb0e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33cb5e2d82544e70adcb8dd2dfbbfb0e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40543 } }
I20260812 05:24:26.014520 2328 leader_election.cc:302] T 00000000000000000000000000000000 P 33cb5e2d82544e70adcb8dd2dfbbfb0e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 33cb5e2d82544e70adcb8dd2dfbbfb0e; no voters:
I20260812 05:24:26.014667 2328 leader_election.cc:288] T 00000000000000000000000000000000 P 33cb5e2d82544e70adcb8dd2dfbbfb0e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 05:24:26.014714 2332 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 33cb5e2d82544e70adcb8dd2dfbbfb0e [term 1 FOLLOWER]: Leader election won for term 1
I20260812 05:24:26.014863 2332 raft_consensus.cc:686] T 00000000000000000000000000000000 P 33cb5e2d82544e70adcb8dd2dfbbfb0e [term 1 LEADER]: Becoming Leader. State: Replica: 33cb5e2d82544e70adcb8dd2dfbbfb0e, State: Running, Role: LEADER
I20260812 05:24:26.014930 2328 sys_catalog.cc:564] T 00000000000000000000000000000000 P 33cb5e2d82544e70adcb8dd2dfbbfb0e [sys.catalog]: configured and running, proceeding with master startup.
I20260812 05:24:26.014986 2332 consensus_queue.cc:237] T 00000000000000000000000000000000 P 33cb5e2d82544e70adcb8dd2dfbbfb0e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33cb5e2d82544e70adcb8dd2dfbbfb0e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40543 } }
I20260812 05:24:26.015502 2339 sys_catalog.cc:455] T 00000000000000000000000000000000 P 33cb5e2d82544e70adcb8dd2dfbbfb0e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "33cb5e2d82544e70adcb8dd2dfbbfb0e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33cb5e2d82544e70adcb8dd2dfbbfb0e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40543 } } }
I20260812 05:24:26.015597 2339 sys_catalog.cc:458] T 00000000000000000000000000000000 P 33cb5e2d82544e70adcb8dd2dfbbfb0e [sys.catalog]: This master's current role is: LEADER
I20260812 05:24:26.015702 2340 sys_catalog.cc:455] T 00000000000000000000000000000000 P 33cb5e2d82544e70adcb8dd2dfbbfb0e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 33cb5e2d82544e70adcb8dd2dfbbfb0e. Latest consensus state: current_term: 1 leader_uuid: "33cb5e2d82544e70adcb8dd2dfbbfb0e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33cb5e2d82544e70adcb8dd2dfbbfb0e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40543 } } }
I20260812 05:24:26.015810 2340 sys_catalog.cc:458] T 00000000000000000000000000000000 P 33cb5e2d82544e70adcb8dd2dfbbfb0e [sys.catalog]: This master's current role is: LEADER
I20260812 05:24:26.015869 32144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0waFvT/build/release/bin/kudu as pid 2221
I20260812 05:24:26.015985 32144 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786512017613366-32144-0/minicluster-data/master-0/wal/instance
I20260812 05:24:26.016119 2351 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 05:24:26.016161 32144 external_mini_cluster.cc:1428] Setting key f303ec1de5af400e0f3cf8a896a24388
I20260812 05:24:26.016692 2351 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 05:24:26.018302 2351 catalog_manager.cc:1347] Generated new cluster ID: 5cf3ebac12f246e2b154ddfd40e446e4
I20260812 05:24:26.018364 2351 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 12 05:24:26 dist-test-slave-42z9 krb5kdc[2202](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512265, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260812 05:24:26.027697 2365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:24:26.016807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38807 (local address 127.0.0.1:40543)
0812 05:24:26.016989 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0812 05:24:26.016997 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 05:24:26.017026 (+ 29us) server_negotiation.cc:407] Connection header received
0812 05:24:26.017237 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:24:26.017246 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:24:26.017301 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:24:26.017464 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:24:26.018192 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:24:26.019585 (+ 1393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:24:26.020715 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:24:26.021319 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:24:26.021410 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:24:26.024190 (+ 2780us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 05:24:26.024194 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 05:24:26.024195 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 05:24:26.024207 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0812 05:24:26.026138 (+ 1931us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:24:26.026500 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:24:26.026503 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:24:26.026503 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:24:26.026620 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:24:26.026792 (+ 172us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:24:26.026793 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:24:26.026793 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:24:26.027109 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 05:24:26.027266 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0812 05:24:26.027332 (+ 66us) server_negotiation.cc:299] Negotiation successful
0812 05:24:26.027467 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":97,"threads_started":1}
I20260812 05:24:26.028795 32144 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 05:24:26.036702 2365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:24:26.030227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34057 (local address 127.0.0.1:40543)
0812 05:24:26.030264 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0812 05:24:26.030266 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 05:24:26.030363 (+ 97us) server_negotiation.cc:407] Connection header received
0812 05:24:26.030401 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:24:26.030402 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:24:26.030411 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:24:26.030484 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:24:26.031004 (+ 520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:24:26.031754 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:24:26.032674 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:24:26.033004 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:24:26.033067 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:24:26.034261 (+ 1194us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 05:24:26.034262 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 05:24:26.034263 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 05:24:26.034269 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0812 05:24:26.035675 (+ 1406us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:24:26.036019 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:24:26.036020 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:24:26.036021 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:24:26.036076 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:24:26.036242 (+ 166us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:24:26.036243 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:24:26.036243 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:24:26.036363 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 05:24:26.036464 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0812 05:24:26.036539 (+ 75us) server_negotiation.cc:299] Negotiation successful
0812 05:24:26.036614 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260812 05:24:26.043747 2351 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 05:24:26.044236 2351 catalog_manager.cc:1504] Loading token signing keys...
I20260812 05:24:26.049489 2351 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 33cb5e2d82544e70adcb8dd2dfbbfb0e: Generated new TSK 0
I20260812 05:24:26.049644 2351 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 05:24:26.055943 32144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0waFvT/build/release/bin/kudu with pid 2221
2026-08-12T05:24:26Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1463 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786512017613366-32144-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 05:24:26 dist-test-slave-42z9 krb5kdc[2381](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 05:24:26 dist-test-slave-42z9 krb5kdc[2381](info): set up 2 sockets
Aug 12 05:24:26 dist-test-slave-42z9 krb5kdc[2381](info): commencing operation
krb5kdc: starting...
W20260812 05:24:27.229859 32144 mini_kdc.cc:121] Time spent starting KDC: real 1.146s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786512017613366-32144-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786512017613366-32144-0/krb5kdc/test-user.keytab.
Aug 12 05:24:27 dist-test-slave-42z9 krb5kdc[2381](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512267, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T05:24:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T05:24:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786512017613366-32144-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786512017613366-32144-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260812 05:24:27.346290 32144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0waFvT/build/release/bin/kudu
/tmp/dist-test-task0waFvT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786512017613366-32144-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786512017613366-32144-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786512017613366-32144-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786512017613366-32144-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42037
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786512017613366-32144-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42879
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42037
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786512017613366-32144-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786512017613366-32144-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786512017613366-32144-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 05:24:27.426566 2410 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 05:24:27.426754 2410 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 05:24:27.426785 2410 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 05:24:27.427899 2410 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 05:24:27.427939 2410 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 05:24:27.427958 2410 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 05:24:27.427974 2410 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 05:24:27.427989 2410 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 05:24:27.429499 2410 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42879
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786512017613366-32144-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786512017613366-32144-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42037
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786512017613366-32144-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42037
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786512017613366-32144-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2410
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786512017613366-32144-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 04:20:34 UTC on ccycloud.quasar-phqdqp.root.comops.site
build id 4581871
I20260812 05:24:27.429740 2410 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 05:24:27.429947 2410 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 05:24:27.434576 2426 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 05:24:27.434594 2424 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 05:24:27.434592 2422 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 05:24:27.434731 2410 server_base.cc:1034] running on GCE node
I20260812 05:24:27.434965 2410 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 05:24:27.435222 2410 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 05:24:27.436365 2410 hybrid_clock.cc:648] HybridClock initialized: now 1786512267436348 us; error 32 us; skew 500 ppm
Aug 12 05:24:27 dist-test-slave-42z9 krb5kdc[2381](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512267, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 05:24:27.442986 2410 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260812 05:24:27.443482 2410 webserver.cc:460] Webserver started at http://127.0.0.1:45899/ using document root <none> and password file <none>
I20260812 05:24:27.443810 2410 fs_manager.cc:359] Metadata directory not provided
I20260812 05:24:27.443881 2410 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 05:24:27.444016 2410 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 05:24:27.445305 2410 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786512017613366-32144-0/minicluster-data/master-0/data/instance:
uuid: "7047573913d148aab94aeaf16a8d476f"
format_stamp: "Formatted at 2026-08-12 05:24:27 on dist-test-slave-42z9"
server_key: "b1786f846d859cbc443c693cdf1a875c"
server_key_iv: "0e1a04bc0c0f62b2c42d3ac7c9cf56f8"
server_key_version: "encryptionkey@0"
I20260812 05:24:27.445652 2410 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786512017613366-32144-0/minicluster-data/master-0/wal/instance:
uuid: "7047573913d148aab94aeaf16a8d476f"
format_stamp: "Formatted at 2026-08-12 05:24:27 on dist-test-slave-42z9"
server_key: "b1786f846d859cbc443c693cdf1a875c"
server_key_iv: "0e1a04bc0c0f62b2c42d3ac7c9cf56f8"
server_key_version: "encryptionkey@0"
I20260812 05:24:27.447180 2410 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260812 05:24:27.447804 2433 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 05:24:27.447943 2410 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260812 05:24:27.448026 2410 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786512017613366-32144-0/minicluster-data/master-0/data,/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786512017613366-32144-0/minicluster-data/master-0/wal
uuid: "7047573913d148aab94aeaf16a8d476f"
format_stamp: "Formatted at 2026-08-12 05:24:27 on dist-test-slave-42z9"
server_key: "b1786f846d859cbc443c693cdf1a875c"
server_key_iv: "0e1a04bc0c0f62b2c42d3ac7c9cf56f8"
server_key_version: "encryptionkey@0"
I20260812 05:24:27.448091 2410 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786512017613366-32144-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786512017613366-32144-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786512017613366-32144-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 05:24:27.465862 2410 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 05:24:27.467561 2410 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 05:24:27.467684 2410 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 05:24:27.471851 2410 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42037
I20260812 05:24:27.471877 2510 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42037 every 8 connection(s)
I20260812 05:24:27.472206 2410 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786512017613366-32144-0/minicluster-data/master-0/data/info.pb
I20260812 05:24:27.473052 2511 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 05:24:27.475278 2511 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7047573913d148aab94aeaf16a8d476f: Bootstrap starting.
I20260812 05:24:27.475834 2511 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7047573913d148aab94aeaf16a8d476f: Neither blocks nor log segments found. Creating new log.
I20260812 05:24:27.476101 2511 log.cc:824] T 00000000000000000000000000000000 P 7047573913d148aab94aeaf16a8d476f: Log is configured to *not* fsync() on all Append() calls
I20260812 05:24:27.476747 2511 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7047573913d148aab94aeaf16a8d476f: No bootstrap required, opened a new log
I20260812 05:24:27.478353 2511 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7047573913d148aab94aeaf16a8d476f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7047573913d148aab94aeaf16a8d476f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42037 } }
I20260812 05:24:27.478487 2511 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7047573913d148aab94aeaf16a8d476f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 05:24:27.478519 2511 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7047573913d148aab94aeaf16a8d476f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7047573913d148aab94aeaf16a8d476f, State: Initialized, Role: FOLLOWER
I20260812 05:24:27.478660 2511 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7047573913d148aab94aeaf16a8d476f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7047573913d148aab94aeaf16a8d476f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42037 } }
I20260812 05:24:27.478736 2511 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7047573913d148aab94aeaf16a8d476f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 05:24:27.478770 2511 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7047573913d148aab94aeaf16a8d476f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 05:24:27.478818 2511 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7047573913d148aab94aeaf16a8d476f [term 0 FOLLOWER]: Advancing to term 1
I20260812 05:24:27.479319 2511 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7047573913d148aab94aeaf16a8d476f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7047573913d148aab94aeaf16a8d476f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42037 } }
I20260812 05:24:27.479422 2511 leader_election.cc:302] T 00000000000000000000000000000000 P 7047573913d148aab94aeaf16a8d476f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7047573913d148aab94aeaf16a8d476f; no voters:
I20260812 05:24:27.479583 2511 leader_election.cc:288] T 00000000000000000000000000000000 P 7047573913d148aab94aeaf16a8d476f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 05:24:27.479624 2514 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7047573913d148aab94aeaf16a8d476f [term 1 FOLLOWER]: Leader election won for term 1
I20260812 05:24:27.479763 2514 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7047573913d148aab94aeaf16a8d476f [term 1 LEADER]: Becoming Leader. State: Replica: 7047573913d148aab94aeaf16a8d476f, State: Running, Role: LEADER
I20260812 05:24:27.479835 2511 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7047573913d148aab94aeaf16a8d476f [sys.catalog]: configured and running, proceeding with master startup.
I20260812 05:24:27.479842 2514 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7047573913d148aab94aeaf16a8d476f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7047573913d148aab94aeaf16a8d476f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42037 } }
I20260812 05:24:27.480176 2514 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7047573913d148aab94aeaf16a8d476f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7047573913d148aab94aeaf16a8d476f. Latest consensus state: current_term: 1 leader_uuid: "7047573913d148aab94aeaf16a8d476f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7047573913d148aab94aeaf16a8d476f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42037 } } }
I20260812 05:24:27.480306 2514 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7047573913d148aab94aeaf16a8d476f [sys.catalog]: This master's current role is: LEADER
I20260812 05:24:27.480559 32144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0waFvT/build/release/bin/kudu as pid 2410
I20260812 05:24:27.480684 32144 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786512017613366-32144-0/minicluster-data/master-0/wal/instance
I20260812 05:24:27.480882 32144 external_mini_cluster.cc:1428] Setting key 9b5245ae47afb6966e164316f530ad76
I20260812 05:24:27.480576 2532 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 05:24:27.480342 2515 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7047573913d148aab94aeaf16a8d476f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7047573913d148aab94aeaf16a8d476f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7047573913d148aab94aeaf16a8d476f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42037 } } }
I20260812 05:24:27.481129 2515 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7047573913d148aab94aeaf16a8d476f [sys.catalog]: This master's current role is: LEADER
I20260812 05:24:27.481513 2532 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 05:24:27.482945 2532 catalog_manager.cc:1347] Generated new cluster ID: 3cef420fa291475394fec2c204033c5c
I20260812 05:24:27.482997 2532 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 05:24:27.487425 2550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:24:27.481304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43603 (local address 127.0.0.1:42037)
0812 05:24:27.481516 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0812 05:24:27.481525 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 05:24:27.481741 (+ 216us) server_negotiation.cc:407] Connection header received
0812 05:24:27.482043 (+ 302us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:24:27.482051 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:24:27.482120 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:24:27.482274 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:24:27.482612 (+ 338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:24:27.483984 (+ 1372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:24:27.485209 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:24:27.485889 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:24:27.486317 (+ 428us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:24:27.486697 (+ 380us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 05:24:27.486701 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 05:24:27.486701 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 05:24:27.486912 (+ 211us) server_negotiation.cc:922] calling sasl_server_start()
0812 05:24:27.486954 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 05:24:27.486969 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 05:24:27.487044 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0812 05:24:27.487118 (+ 74us) server_negotiation.cc:299] Negotiation successful
0812 05:24:27.487188 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160,"thread_start_us":98,"threads_started":1}
I20260812 05:24:27.488519 32144 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 05:24:27.493319 2550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:24:27.490140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56315 (local address 127.0.0.1:42037)
0812 05:24:27.490173 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0812 05:24:27.490175 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 05:24:27.490245 (+ 70us) server_negotiation.cc:407] Connection header received
0812 05:24:27.490294 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:24:27.490295 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:24:27.490321 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:24:27.490382 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:24:27.490501 (+ 119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:24:27.491229 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:24:27.492148 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:24:27.492464 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:24:27.492535 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:24:27.493094 (+ 559us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 05:24:27.493096 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 05:24:27.493096 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 05:24:27.493107 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0812 05:24:27.493122 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 05:24:27.493127 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 05:24:27.493181 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0812 05:24:27.493221 (+ 40us) server_negotiation.cc:299] Negotiation successful
0812 05:24:27.493245 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260812 05:24:27.497821 2532 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 05:24:27.498283 2532 catalog_manager.cc:1504] Loading token signing keys...
I20260812 05:24:27.501907 2532 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7047573913d148aab94aeaf16a8d476f: Generated new TSK 0
I20260812 05:24:27.502061 2532 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 05:24:27.507306 32144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0waFvT/build/release/bin/kudu with pid 2410
2026-08-12T05:24:27Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1449 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786512017613366-32144-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 05:24:27 dist-test-slave-42z9 krb5kdc[2565](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 05:24:27 dist-test-slave-42z9 krb5kdc[2565](info): set up 2 sockets
Aug 12 05:24:27 dist-test-slave-42z9 krb5kdc[2565](info): commencing operation
krb5kdc: starting...
W20260812 05:24:28.669874 32144 mini_kdc.cc:121] Time spent starting KDC: real 1.137s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786512017613366-32144-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786512017613366-32144-0/krb5kdc/test-user.keytab.
Aug 12 05:24:28 dist-test-slave-42z9 krb5kdc[2565](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512268, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T05:24:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T05:24:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786512017613366-32144-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786512017613366-32144-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260812 05:24:28.779760 32144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0waFvT/build/release/bin/kudu
/tmp/dist-test-task0waFvT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786512017613366-32144-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786512017613366-32144-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786512017613366-32144-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786512017613366-32144-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35115
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786512017613366-32144-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44171
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35115
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786512017613366-32144-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786512017613366-32144-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786512017613366-32144-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 05:24:28.859055 2591 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 05:24:28.859269 2591 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 05:24:28.859305 2591 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 05:24:28.860424 2591 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 05:24:28.860466 2591 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 05:24:28.860486 2591 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 05:24:28.860502 2591 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 05:24:28.860518 2591 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 05:24:28.862032 2591 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44171
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786512017613366-32144-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786512017613366-32144-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35115
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786512017613366-32144-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35115
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786512017613366-32144-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2591
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786512017613366-32144-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 04:20:34 UTC on ccycloud.quasar-phqdqp.root.comops.site
build id 4581871
I20260812 05:24:28.862282 2591 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 05:24:28.862483 2591 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 05:24:28.867101 2598 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 05:24:28.867112 2599 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 05:24:28.867195 2591 server_base.cc:1034] running on GCE node
W20260812 05:24:28.867110 2602 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 05:24:28.867482 2591 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 05:24:28.867736 2591 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 05:24:28.868866 2591 hybrid_clock.cc:648] HybridClock initialized: now 1786512268868855 us; error 29 us; skew 500 ppm
Aug 12 05:24:28 dist-test-slave-42z9 krb5kdc[2565](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512268, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 05:24:28.871151 2591 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260812 05:24:28.871510 2591 webserver.cc:460] Webserver started at http://127.0.0.1:42799/ using document root <none> and password file <none>
I20260812 05:24:28.871702 2591 fs_manager.cc:359] Metadata directory not provided
I20260812 05:24:28.871742 2591 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 05:24:28.871829 2591 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 05:24:28.872740 2591 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786512017613366-32144-0/minicluster-data/master-0/data/instance:
uuid: "acf8ffb428db403fb74a3c131dfc6def"
format_stamp: "Formatted at 2026-08-12 05:24:28 on dist-test-slave-42z9"
server_key: "d15076823568bd9ea36e6ec28616a082"
server_key_iv: "20f05b8dd12db8f82e2bbe6f38b815c2"
server_key_version: "encryptionkey@0"
I20260812 05:24:28.873003 2591 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786512017613366-32144-0/minicluster-data/master-0/wal/instance:
uuid: "acf8ffb428db403fb74a3c131dfc6def"
format_stamp: "Formatted at 2026-08-12 05:24:28 on dist-test-slave-42z9"
server_key: "d15076823568bd9ea36e6ec28616a082"
server_key_iv: "20f05b8dd12db8f82e2bbe6f38b815c2"
server_key_version: "encryptionkey@0"
I20260812 05:24:28.874233 2591 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260812 05:24:28.874850 2612 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 05:24:28.875017 2591 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260812 05:24:28.875114 2591 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786512017613366-32144-0/minicluster-data/master-0/data,/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786512017613366-32144-0/minicluster-data/master-0/wal
uuid: "acf8ffb428db403fb74a3c131dfc6def"
format_stamp: "Formatted at 2026-08-12 05:24:28 on dist-test-slave-42z9"
server_key: "d15076823568bd9ea36e6ec28616a082"
server_key_iv: "20f05b8dd12db8f82e2bbe6f38b815c2"
server_key_version: "encryptionkey@0"
I20260812 05:24:28.875176 2591 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786512017613366-32144-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786512017613366-32144-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786512017613366-32144-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 05:24:28.878247 2591 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 05:24:28.879876 2591 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 05:24:28.879990 2591 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 05:24:28.883684 2591 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35115
I20260812 05:24:28.883715 2695 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35115 every 8 connection(s)
I20260812 05:24:28.883993 2591 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786512017613366-32144-0/minicluster-data/master-0/data/info.pb
I20260812 05:24:28.884810 2696 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 05:24:28.886888 2696 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P acf8ffb428db403fb74a3c131dfc6def: Bootstrap starting.
I20260812 05:24:28.887420 2696 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P acf8ffb428db403fb74a3c131dfc6def: Neither blocks nor log segments found. Creating new log.
I20260812 05:24:28.887677 2696 log.cc:824] T 00000000000000000000000000000000 P acf8ffb428db403fb74a3c131dfc6def: Log is configured to *not* fsync() on all Append() calls
I20260812 05:24:28.888298 2696 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P acf8ffb428db403fb74a3c131dfc6def: No bootstrap required, opened a new log
I20260812 05:24:28.889897 2696 raft_consensus.cc:348] T 00000000000000000000000000000000 P acf8ffb428db403fb74a3c131dfc6def [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "acf8ffb428db403fb74a3c131dfc6def" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35115 } }
I20260812 05:24:28.890023 2696 raft_consensus.cc:374] T 00000000000000000000000000000000 P acf8ffb428db403fb74a3c131dfc6def [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 05:24:28.890054 2696 raft_consensus.cc:729] T 00000000000000000000000000000000 P acf8ffb428db403fb74a3c131dfc6def [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acf8ffb428db403fb74a3c131dfc6def, State: Initialized, Role: FOLLOWER
I20260812 05:24:28.890177 2696 consensus_queue.cc:260] T 00000000000000000000000000000000 P acf8ffb428db403fb74a3c131dfc6def [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "acf8ffb428db403fb74a3c131dfc6def" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35115 } }
I20260812 05:24:28.890239 2696 raft_consensus.cc:388] T 00000000000000000000000000000000 P acf8ffb428db403fb74a3c131dfc6def [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 05:24:28.890268 2696 raft_consensus.cc:482] T 00000000000000000000000000000000 P acf8ffb428db403fb74a3c131dfc6def [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 05:24:28.890318 2696 raft_consensus.cc:3037] T 00000000000000000000000000000000 P acf8ffb428db403fb74a3c131dfc6def [term 0 FOLLOWER]: Advancing to term 1
I20260812 05:24:28.890792 2696 raft_consensus.cc:504] T 00000000000000000000000000000000 P acf8ffb428db403fb74a3c131dfc6def [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "acf8ffb428db403fb74a3c131dfc6def" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35115 } }
I20260812 05:24:28.890887 2696 leader_election.cc:302] T 00000000000000000000000000000000 P acf8ffb428db403fb74a3c131dfc6def [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: acf8ffb428db403fb74a3c131dfc6def; no voters:
I20260812 05:24:28.891039 2696 leader_election.cc:288] T 00000000000000000000000000000000 P acf8ffb428db403fb74a3c131dfc6def [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 05:24:28.891073 2700 raft_consensus.cc:2781] T 00000000000000000000000000000000 P acf8ffb428db403fb74a3c131dfc6def [term 1 FOLLOWER]: Leader election won for term 1
I20260812 05:24:28.891216 2700 raft_consensus.cc:686] T 00000000000000000000000000000000 P acf8ffb428db403fb74a3c131dfc6def [term 1 LEADER]: Becoming Leader. State: Replica: acf8ffb428db403fb74a3c131dfc6def, State: Running, Role: LEADER
I20260812 05:24:28.891259 2696 sys_catalog.cc:564] T 00000000000000000000000000000000 P acf8ffb428db403fb74a3c131dfc6def [sys.catalog]: configured and running, proceeding with master startup.
I20260812 05:24:28.891305 2700 consensus_queue.cc:237] T 00000000000000000000000000000000 P acf8ffb428db403fb74a3c131dfc6def [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "acf8ffb428db403fb74a3c131dfc6def" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35115 } }
I20260812 05:24:28.891624 2701 sys_catalog.cc:455] T 00000000000000000000000000000000 P acf8ffb428db403fb74a3c131dfc6def [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "acf8ffb428db403fb74a3c131dfc6def" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "acf8ffb428db403fb74a3c131dfc6def" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35115 } } }
I20260812 05:24:28.891722 2701 sys_catalog.cc:458] T 00000000000000000000000000000000 P acf8ffb428db403fb74a3c131dfc6def [sys.catalog]: This master's current role is: LEADER
I20260812 05:24:28.891638 2702 sys_catalog.cc:455] T 00000000000000000000000000000000 P acf8ffb428db403fb74a3c131dfc6def [sys.catalog]: SysCatalogTable state changed. Reason: New leader acf8ffb428db403fb74a3c131dfc6def. Latest consensus state: current_term: 1 leader_uuid: "acf8ffb428db403fb74a3c131dfc6def" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "acf8ffb428db403fb74a3c131dfc6def" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35115 } } }
I20260812 05:24:28.891800 2702 sys_catalog.cc:458] T 00000000000000000000000000000000 P acf8ffb428db403fb74a3c131dfc6def [sys.catalog]: This master's current role is: LEADER
I20260812 05:24:28.891952 2705 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 05:24:28.892391 2705 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 05:24:28.893729 32144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0waFvT/build/release/bin/kudu as pid 2591
I20260812 05:24:28.893754 2705 catalog_manager.cc:1347] Generated new cluster ID: 35cd6636c34540acae2e7cc29acb11c8
I20260812 05:24:28.893793 2705 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 05:24:28.893860 32144 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786512017613366-32144-0/minicluster-data/master-0/wal/instance
I20260812 05:24:28.894032 32144 external_mini_cluster.cc:1428] Setting key fb7a5ca81f4297b4894444e8ac3c8aa8
I20260812 05:24:28.896301 2728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:24:28.894602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44327 (local address 127.0.0.1:35115)
0812 05:24:28.894911 (+ 309us) server_negotiation.cc:206] Beginning negotiation
0812 05:24:28.894919 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 05:24:28.894948 (+ 29us) server_negotiation.cc:407] Connection header received
0812 05:24:28.895049 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:24:28.895055 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:24:28.895084 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:24:28.895220 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:24:28.895837 (+ 617us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 05:24:28.895840 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 05:24:28.895840 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 05:24:28.895844 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0812 05:24:28.895881 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 05:24:28.895893 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 05:24:28.895947 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0812 05:24:28.896017 (+ 70us) server_negotiation.cc:299] Negotiation successful
0812 05:24:28.896142 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":267,"thread_start_us":238,"threads_started":1}
I20260812 05:24:28.897297 32144 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 05:24:28.899840 2728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:24:28.898765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35015 (local address 127.0.0.1:35115)
0812 05:24:28.898793 (+ 28us) server_negotiation.cc:206] Beginning negotiation
0812 05:24:28.898795 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 05:24:28.899033 (+ 238us) server_negotiation.cc:407] Connection header received
0812 05:24:28.899092 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:24:28.899092 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:24:28.899097 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:24:28.899144 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:24:28.899659 (+ 515us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 05:24:28.899659 (+ 0us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 05:24:28.899660 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 05:24:28.899663 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0812 05:24:28.899675 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 05:24:28.899679 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 05:24:28.899722 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0812 05:24:28.899775 (+ 53us) server_negotiation.cc:299] Negotiation successful
0812 05:24:28.899785 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14}
I20260812 05:24:28.911405 2705 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 05:24:28.911556 2705 catalog_manager.cc:1504] Loading token signing keys...
I20260812 05:24:28.921475 2705 catalog_manager.cc:5868] T 00000000000000000000000000000000 P acf8ffb428db403fb74a3c131dfc6def: Generated new TSK 0
I20260812 05:24:28.921638 2705 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 05:24:28.930032 32144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0waFvT/build/release/bin/kudu with pid 2591
2026-08-12T05:24:28Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1423 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786512017613366-32144-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 05:24:28 dist-test-slave-42z9 krb5kdc[2755](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 05:24:28 dist-test-slave-42z9 krb5kdc[2755](info): set up 2 sockets
Aug 12 05:24:28 dist-test-slave-42z9 krb5kdc[2755](info): commencing operation
krb5kdc: starting...
W20260812 05:24:30.097306 32144 mini_kdc.cc:121] Time spent starting KDC: real 1.141s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786512017613366-32144-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786512017613366-32144-0/krb5kdc/test-user.keytab.
Aug 12 05:24:30 dist-test-slave-42z9 krb5kdc[2755](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512270, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T05:24:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T05:24:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786512017613366-32144-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786512017613366-32144-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260812 05:24:30.208122 32144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0waFvT/build/release/bin/kudu
/tmp/dist-test-task0waFvT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786512017613366-32144-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786512017613366-32144-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786512017613366-32144-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786512017613366-32144-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46393
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786512017613366-32144-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36597
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46393
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786512017613366-32144-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786512017613366-32144-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786512017613366-32144-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 05:24:30.286761 2781 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 05:24:30.286963 2781 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 05:24:30.286993 2781 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 05:24:30.288106 2781 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 05:24:30.288147 2781 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 05:24:30.288165 2781 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 05:24:30.288182 2781 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 05:24:30.288198 2781 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 05:24:30.289674 2781 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36597
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786512017613366-32144-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786512017613366-32144-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46393
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786512017613366-32144-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46393
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786512017613366-32144-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2781
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786512017613366-32144-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 04:20:34 UTC on ccycloud.quasar-phqdqp.root.comops.site
build id 4581871
I20260812 05:24:30.289898 2781 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 05:24:30.290115 2781 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 05:24:30.294844 2794 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 05:24:30.294819 2797 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 05:24:30.294826 2795 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 05:24:30.294868 2781 server_base.cc:1034] running on GCE node
I20260812 05:24:30.295187 2781 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 05:24:30.295455 2781 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 05:24:30.296597 2781 hybrid_clock.cc:648] HybridClock initialized: now 1786512270296579 us; error 34 us; skew 500 ppm
Aug 12 05:24:30 dist-test-slave-42z9 krb5kdc[2755](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512270, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 05:24:30.298913 2781 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260812 05:24:30.299328 2781 webserver.cc:460] Webserver started at http://127.0.0.1:41067/ using document root <none> and password file <none>
I20260812 05:24:30.299556 2781 fs_manager.cc:359] Metadata directory not provided
I20260812 05:24:30.299605 2781 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 05:24:30.299727 2781 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 05:24:30.300709 2781 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786512017613366-32144-0/minicluster-data/master-0/data/instance:
uuid: "20ba967a474f4f23aff941ba07f6452b"
format_stamp: "Formatted at 2026-08-12 05:24:30 on dist-test-slave-42z9"
server_key: "d6ce4d0d40e32f09d1ff223b2a9327b9"
server_key_iv: "eb829a0ea5424620a39114d4f3ca7857"
server_key_version: "encryptionkey@0"
I20260812 05:24:30.301033 2781 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786512017613366-32144-0/minicluster-data/master-0/wal/instance:
uuid: "20ba967a474f4f23aff941ba07f6452b"
format_stamp: "Formatted at 2026-08-12 05:24:30 on dist-test-slave-42z9"
server_key: "d6ce4d0d40e32f09d1ff223b2a9327b9"
server_key_iv: "eb829a0ea5424620a39114d4f3ca7857"
server_key_version: "encryptionkey@0"
I20260812 05:24:30.302264 2781 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260812 05:24:30.302902 2812 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 05:24:30.303051 2781 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260812 05:24:30.303140 2781 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786512017613366-32144-0/minicluster-data/master-0/data,/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786512017613366-32144-0/minicluster-data/master-0/wal
uuid: "20ba967a474f4f23aff941ba07f6452b"
format_stamp: "Formatted at 2026-08-12 05:24:30 on dist-test-slave-42z9"
server_key: "d6ce4d0d40e32f09d1ff223b2a9327b9"
server_key_iv: "eb829a0ea5424620a39114d4f3ca7857"
server_key_version: "encryptionkey@0"
I20260812 05:24:30.303196 2781 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786512017613366-32144-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786512017613366-32144-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786512017613366-32144-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 05:24:30.313048 2781 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 05:24:30.314927 2781 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 05:24:30.315060 2781 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 05:24:30.318934 2781 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46393
I20260812 05:24:30.319002 2896 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46393 every 8 connection(s)
I20260812 05:24:30.319312 2781 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786512017613366-32144-0/minicluster-data/master-0/data/info.pb
I20260812 05:24:30.320145 2897 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 05:24:30.322047 32144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0waFvT/build/release/bin/kudu as pid 2781
I20260812 05:24:30.322150 32144 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786512017613366-32144-0/minicluster-data/master-0/wal/instance
I20260812 05:24:30.322315 32144 external_mini_cluster.cc:1428] Setting key fce467276ac90523fbd5081100b90d93
I20260812 05:24:30.322554 2897 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 20ba967a474f4f23aff941ba07f6452b: Bootstrap starting.
I20260812 05:24:30.323241 2897 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 20ba967a474f4f23aff941ba07f6452b: Neither blocks nor log segments found. Creating new log.
I20260812 05:24:30.323518 2897 log.cc:824] T 00000000000000000000000000000000 P 20ba967a474f4f23aff941ba07f6452b: Log is configured to *not* fsync() on all Append() calls
I20260812 05:24:30.324255 2897 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 20ba967a474f4f23aff941ba07f6452b: No bootstrap required, opened a new log
I20260812 05:24:30.326045 2897 raft_consensus.cc:348] T 00000000000000000000000000000000 P 20ba967a474f4f23aff941ba07f6452b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20ba967a474f4f23aff941ba07f6452b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46393 } }
I20260812 05:24:30.326171 2897 raft_consensus.cc:374] T 00000000000000000000000000000000 P 20ba967a474f4f23aff941ba07f6452b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 05:24:30.326192 2897 raft_consensus.cc:729] T 00000000000000000000000000000000 P 20ba967a474f4f23aff941ba07f6452b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20ba967a474f4f23aff941ba07f6452b, State: Initialized, Role: FOLLOWER
I20260812 05:24:30.326298 2897 consensus_queue.cc:260] T 00000000000000000000000000000000 P 20ba967a474f4f23aff941ba07f6452b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20ba967a474f4f23aff941ba07f6452b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46393 } }
I20260812 05:24:30.326356 2897 raft_consensus.cc:388] T 00000000000000000000000000000000 P 20ba967a474f4f23aff941ba07f6452b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 05:24:30.326376 2897 raft_consensus.cc:482] T 00000000000000000000000000000000 P 20ba967a474f4f23aff941ba07f6452b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 05:24:30.326421 2897 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 20ba967a474f4f23aff941ba07f6452b [term 0 FOLLOWER]: Advancing to term 1
I20260812 05:24:30.326933 2897 raft_consensus.cc:504] T 00000000000000000000000000000000 P 20ba967a474f4f23aff941ba07f6452b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20ba967a474f4f23aff941ba07f6452b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46393 } }
I20260812 05:24:30.327039 2897 leader_election.cc:302] T 00000000000000000000000000000000 P 20ba967a474f4f23aff941ba07f6452b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 20ba967a474f4f23aff941ba07f6452b; no voters:
I20260812 05:24:30.327188 2897 leader_election.cc:288] T 00000000000000000000000000000000 P 20ba967a474f4f23aff941ba07f6452b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 05:24:30.327224 2904 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 20ba967a474f4f23aff941ba07f6452b [term 1 FOLLOWER]: Leader election won for term 1
I20260812 05:24:30.327415 2904 raft_consensus.cc:686] T 00000000000000000000000000000000 P 20ba967a474f4f23aff941ba07f6452b [term 1 LEADER]: Becoming Leader. State: Replica: 20ba967a474f4f23aff941ba07f6452b, State: Running, Role: LEADER
I20260812 05:24:30.327457 2897 sys_catalog.cc:564] T 00000000000000000000000000000000 P 20ba967a474f4f23aff941ba07f6452b [sys.catalog]: configured and running, proceeding with master startup.
I20260812 05:24:30.327514 2904 consensus_queue.cc:237] T 00000000000000000000000000000000 P 20ba967a474f4f23aff941ba07f6452b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20ba967a474f4f23aff941ba07f6452b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46393 } }
I20260812 05:24:30.327979 2906 sys_catalog.cc:455] T 00000000000000000000000000000000 P 20ba967a474f4f23aff941ba07f6452b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "20ba967a474f4f23aff941ba07f6452b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20ba967a474f4f23aff941ba07f6452b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46393 } } }
I20260812 05:24:30.328065 2906 sys_catalog.cc:458] T 00000000000000000000000000000000 P 20ba967a474f4f23aff941ba07f6452b [sys.catalog]: This master's current role is: LEADER
I20260812 05:24:30.328166 2904 sys_catalog.cc:455] T 00000000000000000000000000000000 P 20ba967a474f4f23aff941ba07f6452b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 20ba967a474f4f23aff941ba07f6452b. Latest consensus state: current_term: 1 leader_uuid: "20ba967a474f4f23aff941ba07f6452b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20ba967a474f4f23aff941ba07f6452b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46393 } } }
I20260812 05:24:30.328254 2904 sys_catalog.cc:458] T 00000000000000000000000000000000 P 20ba967a474f4f23aff941ba07f6452b [sys.catalog]: This master's current role is: LEADER
Aug 12 05:24:30 dist-test-slave-42z9 krb5kdc[2755](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512270, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260812 05:24:30.329406 2917 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 05:24:30.329828 2917 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 05:24:30.331252 2917 catalog_manager.cc:1347] Generated new cluster ID: de98f2966617493d98b83aac33776a8e
I20260812 05:24:30.331297 2917 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 05:24:30.332832 2901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:24:30.322852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.33:54197 (local address 127.0.0.1:46393)
0812 05:24:30.323008 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0812 05:24:30.323015 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 05:24:30.323039 (+ 24us) server_negotiation.cc:407] Connection header received
0812 05:24:30.323478 (+ 439us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:24:30.323485 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:24:30.323532 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:24:30.323649 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:24:30.324423 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:24:30.325341 (+ 918us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:24:30.326102 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:24:30.326628 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:24:30.326722 (+ 94us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:24:30.329358 (+ 2636us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 05:24:30.329361 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 05:24:30.329361 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 05:24:30.329369 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0812 05:24:30.331210 (+ 1841us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:24:30.331634 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:24:30.331637 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:24:30.331637 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:24:30.331739 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:24:30.331950 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:24:30.331951 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:24:30.331952 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:24:30.332240 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 05:24:30.332388 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0812 05:24:30.332537 (+ 149us) server_negotiation.cc:299] Negotiation successful
0812 05:24:30.332639 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":65,"threads_started":1}
I20260812 05:24:30.333781 32144 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 05:24:30.337296 2917 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 05:24:30.337967 2917 catalog_manager.cc:1504] Loading token signing keys...
I20260812 05:24:30.344020 2901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:24:30.335219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.33:50067 (local address 127.0.0.1:46393)
0812 05:24:30.335252 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0812 05:24:30.335254 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 05:24:30.337040 (+ 1786us) server_negotiation.cc:407] Connection header received
0812 05:24:30.337131 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:24:30.337132 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:24:30.337142 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:24:30.337199 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:24:30.337852 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:24:30.338617 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:24:30.339620 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:24:30.339947 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:24:30.340010 (+ 63us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:24:30.341283 (+ 1273us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 05:24:30.341285 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 05:24:30.341285 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 05:24:30.341292 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 05:24:30.342877 (+ 1585us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:24:30.343276 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:24:30.343277 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:24:30.343278 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:24:30.343333 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:24:30.343515 (+ 182us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:24:30.343516 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:24:30.343516 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:24:30.343631 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 05:24:30.343735 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0812 05:24:30.343872 (+ 137us) server_negotiation.cc:299] Negotiation successful
0812 05:24:30.343933 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260812 05:24:30.344357 2917 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 20ba967a474f4f23aff941ba07f6452b: Generated new TSK 0
I20260812 05:24:30.344606 2917 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 05:24:30.348614 32144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0waFvT/build/release/bin/kudu with pid 2781
W20260812 05:24:30.362759 2752 connection.cc:535] client connection to 127.0.0.1:46393 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:46393): Connection reset by peer (error 104)
2026-08-12T05:24:30Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1420 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786512017613366-32144-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 05:24:30 dist-test-slave-42z9 krb5kdc[2950](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 05:24:30 dist-test-slave-42z9 krb5kdc[2950](info): set up 2 sockets
Aug 12 05:24:30 dist-test-slave-42z9 krb5kdc[2950](info): commencing operation
krb5kdc: starting...
W20260812 05:24:31.524920 32144 mini_kdc.cc:121] Time spent starting KDC: real 1.149s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786512017613366-32144-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786512017613366-32144-0/krb5kdc/test-user.keytab.
Aug 12 05:24:31 dist-test-slave-42z9 krb5kdc[2950](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512271, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T05:24:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T05:24:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786512017613366-32144-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786512017613366-32144-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260812 05:24:31.636375 32144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0waFvT/build/release/bin/kudu
/tmp/dist-test-task0waFvT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786512017613366-32144-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786512017613366-32144-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786512017613366-32144-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786512017613366-32144-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37459
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786512017613366-32144-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40613
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37459
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786512017613366-32144-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786512017613366-32144-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786512017613366-32144-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 05:24:31.716938 2980 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 05:24:31.717133 2980 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 05:24:31.717155 2980 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 05:24:31.718293 2980 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 05:24:31.718336 2980 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 05:24:31.718349 2980 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 05:24:31.718371 2980 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 05:24:31.718393 2980 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 05:24:31.719856 2980 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40613
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786512017613366-32144-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786512017613366-32144-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37459
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786512017613366-32144-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37459
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786512017613366-32144-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2980
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786512017613366-32144-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 04:20:34 UTC on ccycloud.quasar-phqdqp.root.comops.site
build id 4581871
I20260812 05:24:31.720089 2980 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 05:24:31.720291 2980 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 05:24:31.724819 2986 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 05:24:31.724830 2991 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 05:24:31.724830 2989 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 05:24:31.726455 2980 server_base.cc:1034] running on GCE node
I20260812 05:24:31.726688 2980 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 05:24:31.726943 2980 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 05:24:31.728070 2980 hybrid_clock.cc:648] HybridClock initialized: now 1786512271728058 us; error 31 us; skew 500 ppm
Aug 12 05:24:31 dist-test-slave-42z9 krb5kdc[2950](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512271, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 05:24:31.730468 2980 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260812 05:24:31.730901 2980 webserver.cc:460] Webserver started at http://127.0.0.1:34615/ using document root <none> and password file <none>
I20260812 05:24:31.731112 2980 fs_manager.cc:359] Metadata directory not provided
I20260812 05:24:31.731160 2980 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 05:24:31.731287 2980 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 05:24:31.732334 2980 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786512017613366-32144-0/minicluster-data/master-0/data/instance:
uuid: "a67657c5b119489ba843e83972f10149"
format_stamp: "Formatted at 2026-08-12 05:24:31 on dist-test-slave-42z9"
server_key: "1c5edc2a6a62d7d52cd750b4a91718de"
server_key_iv: "817fd37bbd2a5b2902498dbed838771e"
server_key_version: "encryptionkey@0"
I20260812 05:24:31.732671 2980 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786512017613366-32144-0/minicluster-data/master-0/wal/instance:
uuid: "a67657c5b119489ba843e83972f10149"
format_stamp: "Formatted at 2026-08-12 05:24:31 on dist-test-slave-42z9"
server_key: "1c5edc2a6a62d7d52cd750b4a91718de"
server_key_iv: "817fd37bbd2a5b2902498dbed838771e"
server_key_version: "encryptionkey@0"
I20260812 05:24:31.733932 2980 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260812 05:24:31.734593 2999 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 05:24:31.734740 2980 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260812 05:24:31.734824 2980 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786512017613366-32144-0/minicluster-data/master-0/data,/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786512017613366-32144-0/minicluster-data/master-0/wal
uuid: "a67657c5b119489ba843e83972f10149"
format_stamp: "Formatted at 2026-08-12 05:24:31 on dist-test-slave-42z9"
server_key: "1c5edc2a6a62d7d52cd750b4a91718de"
server_key_iv: "817fd37bbd2a5b2902498dbed838771e"
server_key_version: "encryptionkey@0"
I20260812 05:24:31.734882 2980 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786512017613366-32144-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786512017613366-32144-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786512017613366-32144-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 05:24:31.746690 2980 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 05:24:31.748456 2980 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 05:24:31.748589 2980 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 05:24:31.752506 2980 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37459
I20260812 05:24:31.752537 3089 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37459 every 8 connection(s)
I20260812 05:24:31.752851 2980 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786512017613366-32144-0/minicluster-data/master-0/data/info.pb
I20260812 05:24:31.753727 3091 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 05:24:31.755939 3091 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a67657c5b119489ba843e83972f10149: Bootstrap starting.
I20260812 05:24:31.756496 3091 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a67657c5b119489ba843e83972f10149: Neither blocks nor log segments found. Creating new log.
I20260812 05:24:31.756755 3091 log.cc:824] T 00000000000000000000000000000000 P a67657c5b119489ba843e83972f10149: Log is configured to *not* fsync() on all Append() calls
I20260812 05:24:31.757375 3091 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a67657c5b119489ba843e83972f10149: No bootstrap required, opened a new log
I20260812 05:24:31.758975 3091 raft_consensus.cc:348] T 00000000000000000000000000000000 P a67657c5b119489ba843e83972f10149 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a67657c5b119489ba843e83972f10149" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37459 } }
I20260812 05:24:31.759109 3091 raft_consensus.cc:374] T 00000000000000000000000000000000 P a67657c5b119489ba843e83972f10149 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 05:24:31.759131 3091 raft_consensus.cc:729] T 00000000000000000000000000000000 P a67657c5b119489ba843e83972f10149 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a67657c5b119489ba843e83972f10149, State: Initialized, Role: FOLLOWER
I20260812 05:24:31.759255 3091 consensus_queue.cc:260] T 00000000000000000000000000000000 P a67657c5b119489ba843e83972f10149 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a67657c5b119489ba843e83972f10149" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37459 } }
I20260812 05:24:31.759310 3091 raft_consensus.cc:388] T 00000000000000000000000000000000 P a67657c5b119489ba843e83972f10149 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 05:24:31.759341 3091 raft_consensus.cc:482] T 00000000000000000000000000000000 P a67657c5b119489ba843e83972f10149 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 05:24:31.759398 3091 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a67657c5b119489ba843e83972f10149 [term 0 FOLLOWER]: Advancing to term 1
I20260812 05:24:31.759903 3091 raft_consensus.cc:504] T 00000000000000000000000000000000 P a67657c5b119489ba843e83972f10149 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a67657c5b119489ba843e83972f10149" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37459 } }
I20260812 05:24:31.759995 3091 leader_election.cc:302] T 00000000000000000000000000000000 P a67657c5b119489ba843e83972f10149 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a67657c5b119489ba843e83972f10149; no voters:
I20260812 05:24:31.760139 3091 leader_election.cc:288] T 00000000000000000000000000000000 P a67657c5b119489ba843e83972f10149 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 05:24:31.760176 3094 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a67657c5b119489ba843e83972f10149 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 05:24:31.760324 3094 raft_consensus.cc:686] T 00000000000000000000000000000000 P a67657c5b119489ba843e83972f10149 [term 1 LEADER]: Becoming Leader. State: Replica: a67657c5b119489ba843e83972f10149, State: Running, Role: LEADER
I20260812 05:24:31.760371 3091 sys_catalog.cc:564] T 00000000000000000000000000000000 P a67657c5b119489ba843e83972f10149 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 05:24:31.760413 3094 consensus_queue.cc:237] T 00000000000000000000000000000000 P a67657c5b119489ba843e83972f10149 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a67657c5b119489ba843e83972f10149" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37459 } }
I20260812 05:24:31.760784 3098 sys_catalog.cc:455] T 00000000000000000000000000000000 P a67657c5b119489ba843e83972f10149 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a67657c5b119489ba843e83972f10149. Latest consensus state: current_term: 1 leader_uuid: "a67657c5b119489ba843e83972f10149" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a67657c5b119489ba843e83972f10149" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37459 } } }
I20260812 05:24:31.760870 3098 sys_catalog.cc:458] T 00000000000000000000000000000000 P a67657c5b119489ba843e83972f10149 [sys.catalog]: This master's current role is: LEADER
I20260812 05:24:31.761109 3108 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 05:24:31.761191 32144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0waFvT/build/release/bin/kudu as pid 2980
I20260812 05:24:31.761294 32144 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786512017613366-32144-0/minicluster-data/master-0/wal/instance
I20260812 05:24:31.761534 32144 external_mini_cluster.cc:1428] Setting key 3674f6004048fdff06fd7a9e833d32f4
I20260812 05:24:31.761662 3108 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 05:24:31.760993 3097 sys_catalog.cc:455] T 00000000000000000000000000000000 P a67657c5b119489ba843e83972f10149 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a67657c5b119489ba843e83972f10149" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a67657c5b119489ba843e83972f10149" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37459 } } }
I20260812 05:24:31.762239 3097 sys_catalog.cc:458] T 00000000000000000000000000000000 P a67657c5b119489ba843e83972f10149 [sys.catalog]: This master's current role is: LEADER
I20260812 05:24:31.763352 3108 catalog_manager.cc:1347] Generated new cluster ID: 1c98629aa8384cdfa891283e614ed4c6
I20260812 05:24:31.763406 3108 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 05:24:31.766621 3124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:24:31.762200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.33:51159 (local address 127.0.0.1:37459)
0812 05:24:31.762356 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0812 05:24:31.762365 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 05:24:31.762583 (+ 218us) server_negotiation.cc:407] Connection header received
0812 05:24:31.762719 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:24:31.762726 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:24:31.762782 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:24:31.762923 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:24:31.763128 (+ 205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:24:31.764093 (+ 965us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:24:31.764838 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:24:31.765295 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:24:31.765388 (+ 93us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:24:31.766021 (+ 633us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 05:24:31.766024 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 05:24:31.766025 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 05:24:31.766204 (+ 179us) server_negotiation.cc:922] calling sasl_server_start()
0812 05:24:31.766234 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 05:24:31.766245 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 05:24:31.766313 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0812 05:24:31.766401 (+ 88us) server_negotiation.cc:299] Negotiation successful
0812 05:24:31.766428 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":80,"threads_started":1}
I20260812 05:24:31.767496 32144 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 05:24:31.776012 3124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:24:31.769689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.33:43907 (local address 127.0.0.1:37459)
0812 05:24:31.769727 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0812 05:24:31.769728 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0812 05:24:31.773608 (+ 3880us) server_negotiation.cc:407] Connection header received
0812 05:24:31.773659 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:24:31.773660 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:24:31.773671 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:24:31.773717 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:24:31.773836 (+ 119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:24:31.774305 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:24:31.774977 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:24:31.775186 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:24:31.775238 (+ 52us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:24:31.775785 (+ 547us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 05:24:31.775786 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 05:24:31.775786 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 05:24:31.775790 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0812 05:24:31.775803 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 05:24:31.775807 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 05:24:31.775849 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0812 05:24:31.775924 (+ 75us) server_negotiation.cc:299] Negotiation successful
0812 05:24:31.775935 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260812 05:24:31.780078 3108 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 05:24:31.780578 3108 catalog_manager.cc:1504] Loading token signing keys...
I20260812 05:24:31.791440 3108 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a67657c5b119489ba843e83972f10149: Generated new TSK 0
I20260812 05:24:31.791613 3108 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 05:24:31.794251 32144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0waFvT/build/release/bin/kudu with pid 2980
W20260812 05:24:31.813271 2948 connection.cc:535] client connection to 127.0.0.1:37459 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:37459): Connection reset by peer (error 104)
2026-08-12T05:24:31Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1450 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786512017613366-32144-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 05:24:31 dist-test-slave-42z9 krb5kdc[3136](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 05:24:31 dist-test-slave-42z9 krb5kdc[3136](info): set up 2 sockets
Aug 12 05:24:31 dist-test-slave-42z9 krb5kdc[3136](info): commencing operation
krb5kdc: starting...
W20260812 05:24:32.963750 32144 mini_kdc.cc:121] Time spent starting KDC: real 1.137s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786512017613366-32144-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786512017613366-32144-0/krb5kdc/test-user.keytab.
Aug 12 05:24:33 dist-test-slave-42z9 krb5kdc[3136](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512273, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T05:24:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T05:24:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786512017613366-32144-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786512017613366-32144-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260812 05:24:33.077651 32144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0waFvT/build/release/bin/kudu
/tmp/dist-test-task0waFvT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786512017613366-32144-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786512017613366-32144-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786512017613366-32144-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786512017613366-32144-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41651
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786512017613366-32144-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44957
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41651
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786512017613366-32144-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786512017613366-32144-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786512017613366-32144-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 05:24:33.158911 3158 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 05:24:33.159114 3158 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 05:24:33.159137 3158 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 05:24:33.160254 3158 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 05:24:33.160290 3158 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 05:24:33.160303 3158 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 05:24:33.160315 3158 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 05:24:33.160326 3158 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 05:24:33.161849 3158 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44957
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786512017613366-32144-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786512017613366-32144-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41651
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786512017613366-32144-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41651
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786512017613366-32144-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3158
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786512017613366-32144-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 04:20:34 UTC on ccycloud.quasar-phqdqp.root.comops.site
build id 4581871
I20260812 05:24:33.162050 3158 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 05:24:33.162226 3158 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 05:24:33.166738 3169 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 05:24:33.166744 3168 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 05:24:33.166873 3158 server_base.cc:1034] running on GCE node
W20260812 05:24:33.166744 3171 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 05:24:33.167141 3158 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 05:24:33.167410 3158 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 05:24:33.168552 3158 hybrid_clock.cc:648] HybridClock initialized: now 1786512273168535 us; error 32 us; skew 500 ppm
Aug 12 05:24:33 dist-test-slave-42z9 krb5kdc[3136](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512273, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 05:24:33.180586 3158 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260812 05:24:33.181051 3158 webserver.cc:460] Webserver started at http://127.0.0.1:34039/ using document root <none> and password file <none>
I20260812 05:24:33.181279 3158 fs_manager.cc:359] Metadata directory not provided
I20260812 05:24:33.181329 3158 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 05:24:33.181489 3158 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 05:24:33.182581 3158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786512017613366-32144-0/minicluster-data/master-0/data/instance:
uuid: "da86ec1b24e441dda432aa46587a42c0"
format_stamp: "Formatted at 2026-08-12 05:24:33 on dist-test-slave-42z9"
server_key: "3394d406a40c2d43defb2237f851fcf1"
server_key_iv: "af3782e63cbda427e241b556d66ab6a3"
server_key_version: "encryptionkey@0"
I20260812 05:24:33.182915 3158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786512017613366-32144-0/minicluster-data/master-0/wal/instance:
uuid: "da86ec1b24e441dda432aa46587a42c0"
format_stamp: "Formatted at 2026-08-12 05:24:33 on dist-test-slave-42z9"
server_key: "3394d406a40c2d43defb2237f851fcf1"
server_key_iv: "af3782e63cbda427e241b556d66ab6a3"
server_key_version: "encryptionkey@0"
I20260812 05:24:33.184149 3158 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260812 05:24:33.184798 3183 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 05:24:33.184943 3158 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260812 05:24:33.185031 3158 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786512017613366-32144-0/minicluster-data/master-0/data,/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786512017613366-32144-0/minicluster-data/master-0/wal
uuid: "da86ec1b24e441dda432aa46587a42c0"
format_stamp: "Formatted at 2026-08-12 05:24:33 on dist-test-slave-42z9"
server_key: "3394d406a40c2d43defb2237f851fcf1"
server_key_iv: "af3782e63cbda427e241b556d66ab6a3"
server_key_version: "encryptionkey@0"
I20260812 05:24:33.185093 3158 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786512017613366-32144-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786512017613366-32144-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786512017613366-32144-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 05:24:33.188342 3158 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 05:24:33.190084 3158 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 05:24:33.190219 3158 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 05:24:33.194056 3158 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41651
I20260812 05:24:33.194085 3267 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41651 every 8 connection(s)
I20260812 05:24:33.194386 3158 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786512017613366-32144-0/minicluster-data/master-0/data/info.pb
I20260812 05:24:33.195188 3269 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 05:24:33.197305 3269 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P da86ec1b24e441dda432aa46587a42c0: Bootstrap starting.
I20260812 05:24:33.197885 3269 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P da86ec1b24e441dda432aa46587a42c0: Neither blocks nor log segments found. Creating new log.
I20260812 05:24:33.198134 3269 log.cc:824] T 00000000000000000000000000000000 P da86ec1b24e441dda432aa46587a42c0: Log is configured to *not* fsync() on all Append() calls
I20260812 05:24:33.198751 3269 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P da86ec1b24e441dda432aa46587a42c0: No bootstrap required, opened a new log
I20260812 05:24:33.200295 3269 raft_consensus.cc:348] T 00000000000000000000000000000000 P da86ec1b24e441dda432aa46587a42c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da86ec1b24e441dda432aa46587a42c0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41651 } }
I20260812 05:24:33.200424 3269 raft_consensus.cc:374] T 00000000000000000000000000000000 P da86ec1b24e441dda432aa46587a42c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 05:24:33.200456 3269 raft_consensus.cc:729] T 00000000000000000000000000000000 P da86ec1b24e441dda432aa46587a42c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: da86ec1b24e441dda432aa46587a42c0, State: Initialized, Role: FOLLOWER
I20260812 05:24:33.200592 3269 consensus_queue.cc:260] T 00000000000000000000000000000000 P da86ec1b24e441dda432aa46587a42c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da86ec1b24e441dda432aa46587a42c0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41651 } }
I20260812 05:24:33.200651 3269 raft_consensus.cc:388] T 00000000000000000000000000000000 P da86ec1b24e441dda432aa46587a42c0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 05:24:33.200683 3269 raft_consensus.cc:482] T 00000000000000000000000000000000 P da86ec1b24e441dda432aa46587a42c0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 05:24:33.200728 3269 raft_consensus.cc:3037] T 00000000000000000000000000000000 P da86ec1b24e441dda432aa46587a42c0 [term 0 FOLLOWER]: Advancing to term 1
I20260812 05:24:33.201202 3269 raft_consensus.cc:504] T 00000000000000000000000000000000 P da86ec1b24e441dda432aa46587a42c0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da86ec1b24e441dda432aa46587a42c0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41651 } }
I20260812 05:24:33.201292 3269 leader_election.cc:302] T 00000000000000000000000000000000 P da86ec1b24e441dda432aa46587a42c0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: da86ec1b24e441dda432aa46587a42c0; no voters:
I20260812 05:24:33.201462 3269 leader_election.cc:288] T 00000000000000000000000000000000 P da86ec1b24e441dda432aa46587a42c0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 05:24:33.201498 3272 raft_consensus.cc:2781] T 00000000000000000000000000000000 P da86ec1b24e441dda432aa46587a42c0 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 05:24:33.201660 3272 raft_consensus.cc:686] T 00000000000000000000000000000000 P da86ec1b24e441dda432aa46587a42c0 [term 1 LEADER]: Becoming Leader. State: Replica: da86ec1b24e441dda432aa46587a42c0, State: Running, Role: LEADER
I20260812 05:24:33.201714 3269 sys_catalog.cc:564] T 00000000000000000000000000000000 P da86ec1b24e441dda432aa46587a42c0 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 05:24:33.201749 3272 consensus_queue.cc:237] T 00000000000000000000000000000000 P da86ec1b24e441dda432aa46587a42c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da86ec1b24e441dda432aa46587a42c0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41651 } }
I20260812 05:24:33.202095 3274 sys_catalog.cc:455] T 00000000000000000000000000000000 P da86ec1b24e441dda432aa46587a42c0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader da86ec1b24e441dda432aa46587a42c0. Latest consensus state: current_term: 1 leader_uuid: "da86ec1b24e441dda432aa46587a42c0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da86ec1b24e441dda432aa46587a42c0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41651 } } }
I20260812 05:24:33.202373 32144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0waFvT/build/release/bin/kudu as pid 3158
I20260812 05:24:33.202490 32144 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786512017613366-32144-0/minicluster-data/master-0/wal/instance
I20260812 05:24:33.202725 32144 external_mini_cluster.cc:1428] Setting key 19befe2c8e260769f4d1081dd27bd6db
I20260812 05:24:33.202179 3274 sys_catalog.cc:458] T 00000000000000000000000000000000 P da86ec1b24e441dda432aa46587a42c0 [sys.catalog]: This master's current role is: LEADER
I20260812 05:24:33.202225 3273 sys_catalog.cc:455] T 00000000000000000000000000000000 P da86ec1b24e441dda432aa46587a42c0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "da86ec1b24e441dda432aa46587a42c0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da86ec1b24e441dda432aa46587a42c0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41651 } } }
I20260812 05:24:33.203367 3273 sys_catalog.cc:458] T 00000000000000000000000000000000 P da86ec1b24e441dda432aa46587a42c0 [sys.catalog]: This master's current role is: LEADER
W20260812 05:24:33.203680 3291 catalog_manager.cc:1558] T 00000000000000000000000000000000 P da86ec1b24e441dda432aa46587a42c0: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 05:24:33.203739 3291 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 05:24:33.203805 3297 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 05:24:33.204013 3297 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 05:24:33.204778 3296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:24:33.203167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.33:57975 (local address 127.0.0.1:41651)
0812 05:24:33.203318 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0812 05:24:33.203327 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 05:24:33.203358 (+ 31us) server_negotiation.cc:407] Connection header received
0812 05:24:33.203473 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:24:33.203480 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:24:33.203511 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:24:33.203660 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:24:33.204257 (+ 597us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 05:24:33.204260 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 05:24:33.204261 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 05:24:33.204265 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0812 05:24:33.204319 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 05:24:33.204333 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 05:24:33.204410 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0812 05:24:33.204447 (+ 37us) server_negotiation.cc:299] Negotiation successful
0812 05:24:33.204592 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105,"thread_start_us":51,"threads_started":1}
I20260812 05:24:33.205687 32144 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 05:24:33.207798 3296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:24:33.207014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.33:60129 (local address 127.0.0.1:41651)
0812 05:24:33.207048 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0812 05:24:33.207050 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 05:24:33.207060 (+ 10us) server_negotiation.cc:407] Connection header received
0812 05:24:33.207078 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:24:33.207079 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:24:33.207083 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:24:33.207147 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:24:33.207598 (+ 451us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 05:24:33.207606 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 05:24:33.207606 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 05:24:33.207610 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0812 05:24:33.207623 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 05:24:33.207627 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 05:24:33.207688 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0812 05:24:33.207719 (+ 31us) server_negotiation.cc:299] Negotiation successful
0812 05:24:33.207730 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260812 05:24:33.213145 3297 catalog_manager.cc:1347] Generated new cluster ID: 39f08ddb71d54ddfbc19203bf336509e
I20260812 05:24:33.213191 3297 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 05:24:33.224002 3297 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 05:24:33.224153 3297 catalog_manager.cc:1504] Loading token signing keys...
I20260812 05:24:33.229123 3297 catalog_manager.cc:5868] T 00000000000000000000000000000000 P da86ec1b24e441dda432aa46587a42c0: Generated new TSK 0
I20260812 05:24:33.229262 3297 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 05:24:33.233389 32144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0waFvT/build/release/bin/kudu with pid 3158
2026-08-12T05:24:33Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1435 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786512017613366-32144-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 05:24:33 dist-test-slave-42z9 krb5kdc[3313](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 05:24:33 dist-test-slave-42z9 krb5kdc[3313](info): set up 2 sockets
Aug 12 05:24:33 dist-test-slave-42z9 krb5kdc[3313](info): commencing operation
krb5kdc: starting...
W20260812 05:24:34.396095 32144 mini_kdc.cc:121] Time spent starting KDC: real 1.133s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786512017613366-32144-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786512017613366-32144-0/krb5kdc/test-user.keytab.
Aug 12 05:24:34 dist-test-slave-42z9 krb5kdc[3313](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512274, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T05:24:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T05:24:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786512017613366-32144-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786512017613366-32144-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260812 05:24:34.506002 32144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0waFvT/build/release/bin/kudu
/tmp/dist-test-task0waFvT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786512017613366-32144-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786512017613366-32144-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786512017613366-32144-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786512017613366-32144-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40901
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786512017613366-32144-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39421
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40901
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786512017613366-32144-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786512017613366-32144-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786512017613366-32144-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 05:24:34.584882 3334 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 05:24:34.585083 3334 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 05:24:34.585103 3334 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 05:24:34.586180 3334 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 05:24:34.586215 3334 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 05:24:34.586227 3334 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 05:24:34.586238 3334 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 05:24:34.586251 3334 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 05:24:34.587630 3334 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39421
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786512017613366-32144-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786512017613366-32144-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40901
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786512017613366-32144-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40901
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786512017613366-32144-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3334
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786512017613366-32144-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 04:20:34 UTC on ccycloud.quasar-phqdqp.root.comops.site
build id 4581871
I20260812 05:24:34.587805 3334 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 05:24:34.587975 3334 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 05:24:34.592506 3341 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 05:24:34.592507 3342 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 05:24:34.592536 3344 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 05:24:34.592928 3334 server_base.cc:1034] running on GCE node
I20260812 05:24:34.593149 3334 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 05:24:34.593395 3334 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 05:24:34.594551 3334 hybrid_clock.cc:648] HybridClock initialized: now 1786512274594539 us; error 33 us; skew 500 ppm
Aug 12 05:24:34 dist-test-slave-42z9 krb5kdc[3313](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512274, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 05:24:34.596765 3334 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260812 05:24:34.597150 3334 webserver.cc:460] Webserver started at http://127.0.0.1:42709/ using document root <none> and password file <none>
I20260812 05:24:34.597335 3334 fs_manager.cc:359] Metadata directory not provided
I20260812 05:24:34.597381 3334 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 05:24:34.597534 3334 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 05:24:34.598495 3334 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786512017613366-32144-0/minicluster-data/master-0/data/instance:
uuid: "04cff7c1ff5a497c964d11ec4eadf619"
format_stamp: "Formatted at 2026-08-12 05:24:34 on dist-test-slave-42z9"
server_key: "1d1dbadb80a1d9651de21ea42ec02042"
server_key_iv: "b0fc01b647032ba029642191d632e6aa"
server_key_version: "encryptionkey@0"
I20260812 05:24:34.598809 3334 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786512017613366-32144-0/minicluster-data/master-0/wal/instance:
uuid: "04cff7c1ff5a497c964d11ec4eadf619"
format_stamp: "Formatted at 2026-08-12 05:24:34 on dist-test-slave-42z9"
server_key: "1d1dbadb80a1d9651de21ea42ec02042"
server_key_iv: "b0fc01b647032ba029642191d632e6aa"
server_key_version: "encryptionkey@0"
I20260812 05:24:34.600083 3334 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260812 05:24:34.600711 3359 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 05:24:34.600862 3334 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260812 05:24:34.600944 3334 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786512017613366-32144-0/minicluster-data/master-0/data,/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786512017613366-32144-0/minicluster-data/master-0/wal
uuid: "04cff7c1ff5a497c964d11ec4eadf619"
format_stamp: "Formatted at 2026-08-12 05:24:34 on dist-test-slave-42z9"
server_key: "1d1dbadb80a1d9651de21ea42ec02042"
server_key_iv: "b0fc01b647032ba029642191d632e6aa"
server_key_version: "encryptionkey@0"
I20260812 05:24:34.601001 3334 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786512017613366-32144-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786512017613366-32144-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786512017613366-32144-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 05:24:34.631465 3334 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 05:24:34.633605 3334 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 05:24:34.633741 3334 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 05:24:34.637514 3334 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40901
I20260812 05:24:34.637560 3449 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40901 every 8 connection(s)
I20260812 05:24:34.637897 3334 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786512017613366-32144-0/minicluster-data/master-0/data/info.pb
I20260812 05:24:34.638819 3450 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 05:24:34.640139 32144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0waFvT/build/release/bin/kudu as pid 3334
I20260812 05:24:34.640259 32144 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786512017613366-32144-0/minicluster-data/master-0/wal/instance
I20260812 05:24:34.640442 32144 external_mini_cluster.cc:1428] Setting key 373790f1aa8bf34f37c8348e04ea0a68
I20260812 05:24:34.641589 3450 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 04cff7c1ff5a497c964d11ec4eadf619: Bootstrap starting.
I20260812 05:24:34.642292 3450 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 04cff7c1ff5a497c964d11ec4eadf619: Neither blocks nor log segments found. Creating new log.
I20260812 05:24:34.642594 3450 log.cc:824] T 00000000000000000000000000000000 P 04cff7c1ff5a497c964d11ec4eadf619: Log is configured to *not* fsync() on all Append() calls
I20260812 05:24:34.643337 3450 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 04cff7c1ff5a497c964d11ec4eadf619: No bootstrap required, opened a new log
I20260812 05:24:34.645421 3450 raft_consensus.cc:348] T 00000000000000000000000000000000 P 04cff7c1ff5a497c964d11ec4eadf619 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04cff7c1ff5a497c964d11ec4eadf619" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40901 } }
I20260812 05:24:34.645606 3450 raft_consensus.cc:374] T 00000000000000000000000000000000 P 04cff7c1ff5a497c964d11ec4eadf619 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 05:24:34.645632 3450 raft_consensus.cc:729] T 00000000000000000000000000000000 P 04cff7c1ff5a497c964d11ec4eadf619 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04cff7c1ff5a497c964d11ec4eadf619, State: Initialized, Role: FOLLOWER
I20260812 05:24:34.645751 3450 consensus_queue.cc:260] T 00000000000000000000000000000000 P 04cff7c1ff5a497c964d11ec4eadf619 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04cff7c1ff5a497c964d11ec4eadf619" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40901 } }
I20260812 05:24:34.645830 3450 raft_consensus.cc:388] T 00000000000000000000000000000000 P 04cff7c1ff5a497c964d11ec4eadf619 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 05:24:34.645861 3450 raft_consensus.cc:482] T 00000000000000000000000000000000 P 04cff7c1ff5a497c964d11ec4eadf619 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 05:24:34.645908 3450 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 04cff7c1ff5a497c964d11ec4eadf619 [term 0 FOLLOWER]: Advancing to term 1
I20260812 05:24:34.646596 3450 raft_consensus.cc:504] T 00000000000000000000000000000000 P 04cff7c1ff5a497c964d11ec4eadf619 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04cff7c1ff5a497c964d11ec4eadf619" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40901 } }
I20260812 05:24:34.646709 3450 leader_election.cc:302] T 00000000000000000000000000000000 P 04cff7c1ff5a497c964d11ec4eadf619 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 04cff7c1ff5a497c964d11ec4eadf619; no voters:
I20260812 05:24:34.646886 3450 leader_election.cc:288] T 00000000000000000000000000000000 P 04cff7c1ff5a497c964d11ec4eadf619 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 05:24:34.646925 3456 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 04cff7c1ff5a497c964d11ec4eadf619 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 05:24:34.647075 3456 raft_consensus.cc:686] T 00000000000000000000000000000000 P 04cff7c1ff5a497c964d11ec4eadf619 [term 1 LEADER]: Becoming Leader. State: Replica: 04cff7c1ff5a497c964d11ec4eadf619, State: Running, Role: LEADER
I20260812 05:24:34.647151 3450 sys_catalog.cc:564] T 00000000000000000000000000000000 P 04cff7c1ff5a497c964d11ec4eadf619 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 05:24:34.647187 3456 consensus_queue.cc:237] T 00000000000000000000000000000000 P 04cff7c1ff5a497c964d11ec4eadf619 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04cff7c1ff5a497c964d11ec4eadf619" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40901 } }
Aug 12 05:24:34 dist-test-slave-42z9 krb5kdc[3313](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512274, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260812 05:24:34.647607 3459 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04cff7c1ff5a497c964d11ec4eadf619 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "04cff7c1ff5a497c964d11ec4eadf619" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04cff7c1ff5a497c964d11ec4eadf619" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40901 } } }
I20260812 05:24:34.647816 3459 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04cff7c1ff5a497c964d11ec4eadf619 [sys.catalog]: This master's current role is: LEADER
I20260812 05:24:34.647755 3456 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04cff7c1ff5a497c964d11ec4eadf619 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 04cff7c1ff5a497c964d11ec4eadf619. Latest consensus state: current_term: 1 leader_uuid: "04cff7c1ff5a497c964d11ec4eadf619" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04cff7c1ff5a497c964d11ec4eadf619" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40901 } } }
I20260812 05:24:34.648299 3456 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04cff7c1ff5a497c964d11ec4eadf619 [sys.catalog]: This master's current role is: LEADER
W20260812 05:24:34.649111 3476 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 04cff7c1ff5a497c964d11ec4eadf619: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 05:24:34.649165 3476 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 05:24:34.649211 3466 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 05:24:34.649369 3466 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 05:24:34.650943 3466 catalog_manager.cc:1347] Generated new cluster ID: b36e419cb082405c9fb83d61feb8bd3e
I20260812 05:24:34.650995 3466 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 05:24:34.652158 3454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:24:34.640945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.33:59159 (local address 127.0.0.1:40901)
0812 05:24:34.641143 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0812 05:24:34.641154 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0812 05:24:34.641193 (+ 39us) server_negotiation.cc:407] Connection header received
0812 05:24:34.641752 (+ 559us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:24:34.641762 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:24:34.641822 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:24:34.641980 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:24:34.642552 (+ 572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:24:34.643856 (+ 1304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:24:34.644591 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:24:34.645221 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:24:34.645334 (+ 113us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:24:34.648079 (+ 2745us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 05:24:34.648083 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 05:24:34.648084 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 05:24:34.648096 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0812 05:24:34.650272 (+ 2176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:24:34.650711 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:24:34.650714 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:24:34.650715 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:24:34.650811 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:24:34.651058 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:24:34.651059 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:24:34.651059 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:24:34.651335 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 05:24:34.651462 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0812 05:24:34.651867 (+ 405us) server_negotiation.cc:299] Negotiation successful
0812 05:24:34.651954 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140,"thread_start_us":90,"threads_started":1}
I20260812 05:24:34.653002 32144 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 05:24:34.662670 3466 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 05:24:34.663128 3466 catalog_manager.cc:1504] Loading token signing keys...
I20260812 05:24:34.669610 3454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:24:34.657823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.33:33095 (local address 127.0.0.1:40901)
0812 05:24:34.657854 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0812 05:24:34.657855 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0812 05:24:34.662405 (+ 4550us) server_negotiation.cc:407] Connection header received
0812 05:24:34.662451 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:24:34.662452 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:24:34.662462 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:24:34.662506 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:24:34.663665 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:24:34.664352 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:24:34.665349 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:24:34.665681 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:24:34.665733 (+ 52us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:24:34.667021 (+ 1288us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 05:24:34.667023 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 05:24:34.667023 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 05:24:34.667029 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0812 05:24:34.668417 (+ 1388us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:24:34.668805 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:24:34.668805 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:24:34.668805 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:24:34.668854 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:24:34.669053 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:24:34.669053 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:24:34.669054 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:24:34.669155 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 05:24:34.669238 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0812 05:24:34.669424 (+ 186us) server_negotiation.cc:299] Negotiation successful
0812 05:24:34.669521 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260812 05:24:34.670717 3466 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 04cff7c1ff5a497c964d11ec4eadf619: Generated new TSK 0
I20260812 05:24:34.670885 3466 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 05:24:34.672919 32144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0waFvT/build/release/bin/kudu with pid 3334
W20260812 05:24:34.684701 3309 connection.cc:535] client connection to 127.0.0.1:40901 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40901): Connection reset by peer (error 104)
2026-08-12T05:24:34Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1435 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786512017613366-32144-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 05:24:34 dist-test-slave-42z9 krb5kdc[3492](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 05:24:34 dist-test-slave-42z9 krb5kdc[3492](info): set up 2 sockets
Aug 12 05:24:34 dist-test-slave-42z9 krb5kdc[3492](info): commencing operation
krb5kdc: starting...
W20260812 05:24:35.830514 32144 mini_kdc.cc:121] Time spent starting KDC: real 1.132s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786512017613366-32144-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786512017613366-32144-0/krb5kdc/test-user.keytab.
Aug 12 05:24:35 dist-test-slave-42z9 krb5kdc[3492](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512275, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T05:24:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T05:24:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786512017613366-32144-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786512017613366-32144-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260812 05:24:35.943702 32144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0waFvT/build/release/bin/kudu
/tmp/dist-test-task0waFvT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786512017613366-32144-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786512017613366-32144-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786512017613366-32144-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786512017613366-32144-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43985
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786512017613366-32144-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33273
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43985
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786512017613366-32144-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786512017613366-32144-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786512017613366-32144-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 05:24:36.025197 3512 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 05:24:36.025390 3512 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 05:24:36.025420 3512 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 05:24:36.026667 3512 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 05:24:36.026722 3512 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 05:24:36.026736 3512 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 05:24:36.026757 3512 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 05:24:36.026784 3512 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 05:24:36.028420 3512 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33273
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786512017613366-32144-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786512017613366-32144-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43985
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786512017613366-32144-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43985
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786512017613366-32144-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3512
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786512017613366-32144-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 04:20:34 UTC on ccycloud.quasar-phqdqp.root.comops.site
build id 4581871
I20260812 05:24:36.028650 3512 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 05:24:36.028861 3512 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 05:24:36.033720 3521 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 05:24:36.033730 3520 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 05:24:36.033730 3525 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 05:24:36.033882 3512 server_base.cc:1034] running on GCE node
I20260812 05:24:36.034108 3512 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 05:24:36.034400 3512 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 05:24:36.035594 3512 hybrid_clock.cc:648] HybridClock initialized: now 1786512276035573 us; error 30 us; skew 500 ppm
Aug 12 05:24:36 dist-test-slave-42z9 krb5kdc[3492](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512276, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 05:24:36.038093 3512 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260812 05:24:36.038573 3512 webserver.cc:460] Webserver started at http://127.0.0.1:43837/ using document root <none> and password file <none>
I20260812 05:24:36.038800 3512 fs_manager.cc:359] Metadata directory not provided
I20260812 05:24:36.038849 3512 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 05:24:36.038969 3512 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 05:24:36.040145 3512 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786512017613366-32144-0/minicluster-data/master-0/data/instance:
uuid: "21d29e6d9c3643c2a6cf7b985b4fc47a"
format_stamp: "Formatted at 2026-08-12 05:24:36 on dist-test-slave-42z9"
server_key: "2326a549c26fcf025fc568d19168255e"
server_key_iv: "789d8ebce14ccedd544789633539a2ff"
server_key_version: "encryptionkey@0"
I20260812 05:24:36.040503 3512 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786512017613366-32144-0/minicluster-data/master-0/wal/instance:
uuid: "21d29e6d9c3643c2a6cf7b985b4fc47a"
format_stamp: "Formatted at 2026-08-12 05:24:36 on dist-test-slave-42z9"
server_key: "2326a549c26fcf025fc568d19168255e"
server_key_iv: "789d8ebce14ccedd544789633539a2ff"
server_key_version: "encryptionkey@0"
I20260812 05:24:36.041811 3512 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260812 05:24:36.042508 3539 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 05:24:36.042659 3512 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260812 05:24:36.042747 3512 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786512017613366-32144-0/minicluster-data/master-0/data,/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786512017613366-32144-0/minicluster-data/master-0/wal
uuid: "21d29e6d9c3643c2a6cf7b985b4fc47a"
format_stamp: "Formatted at 2026-08-12 05:24:36 on dist-test-slave-42z9"
server_key: "2326a549c26fcf025fc568d19168255e"
server_key_iv: "789d8ebce14ccedd544789633539a2ff"
server_key_version: "encryptionkey@0"
I20260812 05:24:36.042804 3512 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786512017613366-32144-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786512017613366-32144-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786512017613366-32144-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 05:24:36.050387 3512 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 05:24:36.052021 3512 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 05:24:36.052141 3512 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 05:24:36.055968 3512 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43985
I20260812 05:24:36.055998 3630 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43985 every 8 connection(s)
I20260812 05:24:36.056309 3512 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786512017613366-32144-0/minicluster-data/master-0/data/info.pb
I20260812 05:24:36.057099 3631 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 05:24:36.058641 32144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0waFvT/build/release/bin/kudu as pid 3512
I20260812 05:24:36.058779 32144 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786512017613366-32144-0/minicluster-data/master-0/wal/instance
I20260812 05:24:36.058990 32144 external_mini_cluster.cc:1428] Setting key 090c8f63e845e52875ef42fbbb420f74
I20260812 05:24:36.059700 3631 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 21d29e6d9c3643c2a6cf7b985b4fc47a: Bootstrap starting.
I20260812 05:24:36.060285 3631 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 21d29e6d9c3643c2a6cf7b985b4fc47a: Neither blocks nor log segments found. Creating new log.
I20260812 05:24:36.060621 3631 log.cc:824] T 00000000000000000000000000000000 P 21d29e6d9c3643c2a6cf7b985b4fc47a: Log is configured to *not* fsync() on all Append() calls
I20260812 05:24:36.061313 3631 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 21d29e6d9c3643c2a6cf7b985b4fc47a: No bootstrap required, opened a new log
I20260812 05:24:36.063514 3631 raft_consensus.cc:348] T 00000000000000000000000000000000 P 21d29e6d9c3643c2a6cf7b985b4fc47a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21d29e6d9c3643c2a6cf7b985b4fc47a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43985 } }
I20260812 05:24:36.063659 3631 raft_consensus.cc:374] T 00000000000000000000000000000000 P 21d29e6d9c3643c2a6cf7b985b4fc47a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 05:24:36.063691 3631 raft_consensus.cc:729] T 00000000000000000000000000000000 P 21d29e6d9c3643c2a6cf7b985b4fc47a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 21d29e6d9c3643c2a6cf7b985b4fc47a, State: Initialized, Role: FOLLOWER
I20260812 05:24:36.063813 3631 consensus_queue.cc:260] T 00000000000000000000000000000000 P 21d29e6d9c3643c2a6cf7b985b4fc47a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21d29e6d9c3643c2a6cf7b985b4fc47a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43985 } }
I20260812 05:24:36.063897 3631 raft_consensus.cc:388] T 00000000000000000000000000000000 P 21d29e6d9c3643c2a6cf7b985b4fc47a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 05:24:36.063936 3631 raft_consensus.cc:482] T 00000000000000000000000000000000 P 21d29e6d9c3643c2a6cf7b985b4fc47a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 05:24:36.063974 3631 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 21d29e6d9c3643c2a6cf7b985b4fc47a [term 0 FOLLOWER]: Advancing to term 1
I20260812 05:24:36.064567 3631 raft_consensus.cc:504] T 00000000000000000000000000000000 P 21d29e6d9c3643c2a6cf7b985b4fc47a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21d29e6d9c3643c2a6cf7b985b4fc47a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43985 } }
I20260812 05:24:36.064615 3635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:24:36.059500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.33:43415 (local address 127.0.0.1:43985)
0812 05:24:36.059662 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0812 05:24:36.059670 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 05:24:36.059702 (+ 32us) server_negotiation.cc:407] Connection header received
0812 05:24:36.060136 (+ 434us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:24:36.060143 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:24:36.060191 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:24:36.060303 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:24:36.060546 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:24:36.061693 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:24:36.062738 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:24:36.063132 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:24:36.063213 (+ 81us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:24:36.063984 (+ 771us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 05:24:36.063989 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 05:24:36.063989 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 05:24:36.064168 (+ 179us) server_negotiation.cc:922] calling sasl_server_start()
0812 05:24:36.064199 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 05:24:36.064211 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 05:24:36.064276 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0812 05:24:36.064392 (+ 116us) server_negotiation.cc:299] Negotiation successful
0812 05:24:36.064416 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":67,"threads_started":1}
I20260812 05:24:36.064658 3631 leader_election.cc:302] T 00000000000000000000000000000000 P 21d29e6d9c3643c2a6cf7b985b4fc47a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 21d29e6d9c3643c2a6cf7b985b4fc47a; no voters:
I20260812 05:24:36.064797 3631 leader_election.cc:288] T 00000000000000000000000000000000 P 21d29e6d9c3643c2a6cf7b985b4fc47a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 05:24:36.064868 3639 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 21d29e6d9c3643c2a6cf7b985b4fc47a [term 1 FOLLOWER]: Leader election won for term 1
I20260812 05:24:36.065016 3639 raft_consensus.cc:686] T 00000000000000000000000000000000 P 21d29e6d9c3643c2a6cf7b985b4fc47a [term 1 LEADER]: Becoming Leader. State: Replica: 21d29e6d9c3643c2a6cf7b985b4fc47a, State: Running, Role: LEADER
I20260812 05:24:36.065029 3631 sys_catalog.cc:564] T 00000000000000000000000000000000 P 21d29e6d9c3643c2a6cf7b985b4fc47a [sys.catalog]: configured and running, proceeding with master startup.
I20260812 05:24:36.065122 3639 consensus_queue.cc:237] T 00000000000000000000000000000000 P 21d29e6d9c3643c2a6cf7b985b4fc47a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21d29e6d9c3643c2a6cf7b985b4fc47a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43985 } }
I20260812 05:24:36.065596 32144 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 05:24:36.065608 3640 sys_catalog.cc:455] T 00000000000000000000000000000000 P 21d29e6d9c3643c2a6cf7b985b4fc47a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 21d29e6d9c3643c2a6cf7b985b4fc47a. Latest consensus state: current_term: 1 leader_uuid: "21d29e6d9c3643c2a6cf7b985b4fc47a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21d29e6d9c3643c2a6cf7b985b4fc47a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43985 } } }
I20260812 05:24:36.065685 3640 sys_catalog.cc:458] T 00000000000000000000000000000000 P 21d29e6d9c3643c2a6cf7b985b4fc47a [sys.catalog]: This master's current role is: LEADER
I20260812 05:24:36.066143 3641 sys_catalog.cc:455] T 00000000000000000000000000000000 P 21d29e6d9c3643c2a6cf7b985b4fc47a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "21d29e6d9c3643c2a6cf7b985b4fc47a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21d29e6d9c3643c2a6cf7b985b4fc47a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43985 } } }
I20260812 05:24:36.066217 3641 sys_catalog.cc:458] T 00000000000000000000000000000000 P 21d29e6d9c3643c2a6cf7b985b4fc47a [sys.catalog]: This master's current role is: LEADER
I20260812 05:24:36.066304 3649 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 05:24:36.066946 3649 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 05:24:36.068995 3649 catalog_manager.cc:1347] Generated new cluster ID: 462167fb50f74ebd82645ec3f1501f54
I20260812 05:24:36.069052 3649 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 05:24:36.069826 3635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:24:36.067073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.33:46123 (local address 127.0.0.1:43985)
0812 05:24:36.067105 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0812 05:24:36.067107 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 05:24:36.067203 (+ 96us) server_negotiation.cc:407] Connection header received
0812 05:24:36.067257 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:24:36.067258 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:24:36.067270 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:24:36.067322 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:24:36.067445 (+ 123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:24:36.067897 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:24:36.068641 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:24:36.068904 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:24:36.068964 (+ 60us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:24:36.069614 (+ 650us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 05:24:36.069615 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 05:24:36.069615 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 05:24:36.069619 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0812 05:24:36.069633 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 05:24:36.069637 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 05:24:36.069686 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0812 05:24:36.069749 (+ 63us) server_negotiation.cc:299] Negotiation successful
0812 05:24:36.069758 (+ 9us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260812 05:24:36.085265 3649 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 05:24:36.085804 3649 catalog_manager.cc:1504] Loading token signing keys...
I20260812 05:24:36.092224 3649 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 21d29e6d9c3643c2a6cf7b985b4fc47a: Generated new TSK 0
I20260812 05:24:36.092394 3649 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 05:24:36.094554 32144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0waFvT/build/release/bin/kudu with pid 3512
W20260812 05:24:36.107059 3490 connection.cc:535] client connection to 127.0.0.1:43985 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:43985): Connection reset by peer (error 104)
2026-08-12T05:24:36Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1422 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786512017613366-32144-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 05:24:36 dist-test-slave-42z9 krb5kdc[3675](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 05:24:36 dist-test-slave-42z9 krb5kdc[3675](info): set up 2 sockets
Aug 12 05:24:36 dist-test-slave-42z9 krb5kdc[3675](info): commencing operation
krb5kdc: starting...
W20260812 05:24:37.268548 32144 mini_kdc.cc:121] Time spent starting KDC: real 1.148s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786512017613366-32144-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786512017613366-32144-0/krb5kdc/test-user.keytab.
Aug 12 05:24:37 dist-test-slave-42z9 krb5kdc[3675](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512277, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T05:24:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T05:24:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786512017613366-32144-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786512017613366-32144-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260812 05:24:37.383652 32144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0waFvT/build/release/bin/kudu
/tmp/dist-test-task0waFvT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786512017613366-32144-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786512017613366-32144-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786512017613366-32144-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786512017613366-32144-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41059
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786512017613366-32144-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46825
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41059
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786512017613366-32144-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786512017613366-32144-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786512017613366-32144-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 05:24:37.463003 3696 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 05:24:37.463208 3696 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 05:24:37.463238 3696 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 05:24:37.464360 3696 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 05:24:37.464401 3696 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 05:24:37.464419 3696 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 05:24:37.464435 3696 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 05:24:37.464449 3696 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 05:24:37.465914 3696 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46825
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786512017613366-32144-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786512017613366-32144-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41059
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786512017613366-32144-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41059
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786512017613366-32144-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3696
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786512017613366-32144-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 04:20:34 UTC on ccycloud.quasar-phqdqp.root.comops.site
build id 4581871
I20260812 05:24:37.466161 3696 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 05:24:37.466357 3696 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 05:24:37.471050 3706 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 05:24:37.471129 3696 server_base.cc:1034] running on GCE node
W20260812 05:24:37.471235 3710 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 05:24:37.471135 3705 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 05:24:37.471544 3696 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 05:24:37.471824 3696 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 05:24:37.472971 3696 hybrid_clock.cc:648] HybridClock initialized: now 1786512277472953 us; error 32 us; skew 500 ppm
Aug 12 05:24:37 dist-test-slave-42z9 krb5kdc[3675](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512277, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 05:24:37.475593 3696 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260812 05:24:37.476095 3696 webserver.cc:460] Webserver started at http://127.0.0.1:46375/ using document root <none> and password file <none>
I20260812 05:24:37.476329 3696 fs_manager.cc:359] Metadata directory not provided
I20260812 05:24:37.476436 3696 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 05:24:37.476598 3696 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 05:24:37.477928 3696 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786512017613366-32144-0/minicluster-data/master-0/data/instance:
uuid: "3bff1ec3a5a743d8a8cfe6b770b14be3"
format_stamp: "Formatted at 2026-08-12 05:24:37 on dist-test-slave-42z9"
server_key: "6c17d0de47ca8e67cd9b76cac923a164"
server_key_iv: "6cf78f8e28204c0a05b8978175d579cc"
server_key_version: "encryptionkey@0"
I20260812 05:24:37.478256 3696 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786512017613366-32144-0/minicluster-data/master-0/wal/instance:
uuid: "3bff1ec3a5a743d8a8cfe6b770b14be3"
format_stamp: "Formatted at 2026-08-12 05:24:37 on dist-test-slave-42z9"
server_key: "6c17d0de47ca8e67cd9b76cac923a164"
server_key_iv: "6cf78f8e28204c0a05b8978175d579cc"
server_key_version: "encryptionkey@0"
I20260812 05:24:37.479471 3696 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260812 05:24:37.480119 3723 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 05:24:37.480261 3696 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260812 05:24:37.480345 3696 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786512017613366-32144-0/minicluster-data/master-0/data,/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786512017613366-32144-0/minicluster-data/master-0/wal
uuid: "3bff1ec3a5a743d8a8cfe6b770b14be3"
format_stamp: "Formatted at 2026-08-12 05:24:37 on dist-test-slave-42z9"
server_key: "6c17d0de47ca8e67cd9b76cac923a164"
server_key_iv: "6cf78f8e28204c0a05b8978175d579cc"
server_key_version: "encryptionkey@0"
I20260812 05:24:37.480412 3696 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786512017613366-32144-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786512017613366-32144-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786512017613366-32144-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 05:24:37.483597 3696 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 05:24:37.485679 3696 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 05:24:37.485798 3696 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 05:24:37.489856 3696 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41059
I20260812 05:24:37.489885 3805 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41059 every 8 connection(s)
I20260812 05:24:37.490195 3696 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786512017613366-32144-0/minicluster-data/master-0/data/info.pb
I20260812 05:24:37.491021 3806 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 05:24:37.493153 3806 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3bff1ec3a5a743d8a8cfe6b770b14be3: Bootstrap starting.
I20260812 05:24:37.493717 3806 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3bff1ec3a5a743d8a8cfe6b770b14be3: Neither blocks nor log segments found. Creating new log.
I20260812 05:24:37.493942 3806 log.cc:824] T 00000000000000000000000000000000 P 3bff1ec3a5a743d8a8cfe6b770b14be3: Log is configured to *not* fsync() on all Append() calls
I20260812 05:24:37.494549 3806 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3bff1ec3a5a743d8a8cfe6b770b14be3: No bootstrap required, opened a new log
I20260812 05:24:37.496050 3806 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3bff1ec3a5a743d8a8cfe6b770b14be3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3bff1ec3a5a743d8a8cfe6b770b14be3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41059 } }
I20260812 05:24:37.496177 3806 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3bff1ec3a5a743d8a8cfe6b770b14be3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 05:24:37.496205 3806 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3bff1ec3a5a743d8a8cfe6b770b14be3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3bff1ec3a5a743d8a8cfe6b770b14be3, State: Initialized, Role: FOLLOWER
I20260812 05:24:37.496304 3806 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3bff1ec3a5a743d8a8cfe6b770b14be3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3bff1ec3a5a743d8a8cfe6b770b14be3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41059 } }
I20260812 05:24:37.496376 3806 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3bff1ec3a5a743d8a8cfe6b770b14be3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 05:24:37.496402 3806 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3bff1ec3a5a743d8a8cfe6b770b14be3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 05:24:37.496433 3806 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3bff1ec3a5a743d8a8cfe6b770b14be3 [term 0 FOLLOWER]: Advancing to term 1
I20260812 05:24:37.496883 3806 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3bff1ec3a5a743d8a8cfe6b770b14be3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3bff1ec3a5a743d8a8cfe6b770b14be3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41059 } }
I20260812 05:24:37.496964 3806 leader_election.cc:302] T 00000000000000000000000000000000 P 3bff1ec3a5a743d8a8cfe6b770b14be3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3bff1ec3a5a743d8a8cfe6b770b14be3; no voters:
I20260812 05:24:37.497081 3806 leader_election.cc:288] T 00000000000000000000000000000000 P 3bff1ec3a5a743d8a8cfe6b770b14be3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 05:24:37.497125 3811 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3bff1ec3a5a743d8a8cfe6b770b14be3 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 05:24:37.497290 3806 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3bff1ec3a5a743d8a8cfe6b770b14be3 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 05:24:37.497306 3811 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3bff1ec3a5a743d8a8cfe6b770b14be3 [term 1 LEADER]: Becoming Leader. State: Replica: 3bff1ec3a5a743d8a8cfe6b770b14be3, State: Running, Role: LEADER
I20260812 05:24:37.497387 3811 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3bff1ec3a5a743d8a8cfe6b770b14be3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3bff1ec3a5a743d8a8cfe6b770b14be3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41059 } }
I20260812 05:24:37.497891 32144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0waFvT/build/release/bin/kudu as pid 3696
I20260812 05:24:37.497910 3816 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3bff1ec3a5a743d8a8cfe6b770b14be3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3bff1ec3a5a743d8a8cfe6b770b14be3. Latest consensus state: current_term: 1 leader_uuid: "3bff1ec3a5a743d8a8cfe6b770b14be3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3bff1ec3a5a743d8a8cfe6b770b14be3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41059 } } }
I20260812 05:24:37.497980 3816 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3bff1ec3a5a743d8a8cfe6b770b14be3 [sys.catalog]: This master's current role is: LEADER
I20260812 05:24:37.498025 32144 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786512017613366-32144-0/minicluster-data/master-0/wal/instance
I20260812 05:24:37.498232 32144 external_mini_cluster.cc:1428] Setting key 463dfaf46de0a44de7b15ce0e3098b4e
I20260812 05:24:37.498494 3828 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 05:24:37.498939 3812 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3bff1ec3a5a743d8a8cfe6b770b14be3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3bff1ec3a5a743d8a8cfe6b770b14be3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3bff1ec3a5a743d8a8cfe6b770b14be3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41059 } } }
I20260812 05:24:37.498996 3828 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 05:24:37.499512 3812 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3bff1ec3a5a743d8a8cfe6b770b14be3 [sys.catalog]: This master's current role is: LEADER
I20260812 05:24:37.500959 3844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:24:37.498850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.33:38755 (local address 127.0.0.1:41059)
0812 05:24:37.499012 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0812 05:24:37.499021 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 05:24:37.499745 (+ 724us) server_negotiation.cc:407] Connection header received
0812 05:24:37.499860 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:24:37.499866 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:24:37.499888 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:24:37.499999 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:24:37.500584 (+ 585us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 05:24:37.500597 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 05:24:37.500598 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 05:24:37.500602 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0812 05:24:37.500643 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 05:24:37.500653 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 05:24:37.500709 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0812 05:24:37.500778 (+ 69us) server_negotiation.cc:299] Negotiation successful
0812 05:24:37.500819 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108,"thread_start_us":66,"threads_started":1}
I20260812 05:24:37.501849 32144 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 05:24:37.503913 3844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:24:37.503119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.33:42513 (local address 127.0.0.1:41059)
0812 05:24:37.503154 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0812 05:24:37.503156 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 05:24:37.503205 (+ 49us) server_negotiation.cc:407] Connection header received
0812 05:24:37.503279 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:24:37.503279 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:24:37.503283 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:24:37.503325 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:24:37.503751 (+ 426us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 05:24:37.503752 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 05:24:37.503752 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 05:24:37.503755 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0812 05:24:37.503766 (+ 11us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 05:24:37.503769 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 05:24:37.503806 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0812 05:24:37.503856 (+ 50us) server_negotiation.cc:299] Negotiation successful
0812 05:24:37.503866 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260812 05:24:37.506317 3828 catalog_manager.cc:1347] Generated new cluster ID: 6fe2122a3cf94d9d96c17cd84124f5f2
I20260812 05:24:37.506384 3828 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 05:24:37.512749 3828 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 05:24:37.512899 3828 catalog_manager.cc:1504] Loading token signing keys...
I20260812 05:24:37.517881 3828 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3bff1ec3a5a743d8a8cfe6b770b14be3: Generated new TSK 0
I20260812 05:24:37.518028 3828 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 05:24:37.519621 32144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0waFvT/build/release/bin/kudu with pid 3696
2026-08-12T05:24:37Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1427 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786512017613366-32144-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 05:24:37 dist-test-slave-42z9 krb5kdc[3858](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 05:24:37 dist-test-slave-42z9 krb5kdc[3858](info): set up 2 sockets
Aug 12 05:24:37 dist-test-slave-42z9 krb5kdc[3858](info): commencing operation
krb5kdc: starting...
W20260812 05:24:38.683290 32144 mini_kdc.cc:121] Time spent starting KDC: real 1.135s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786512017613366-32144-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786512017613366-32144-0/krb5kdc/test-user.keytab.
Aug 12 05:24:38 dist-test-slave-42z9 krb5kdc[3858](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512278, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T05:24:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T05:24:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.100.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.100.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.100.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.62.keytab.
Entry for principal kudu/127.31.100.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.62.keytab.
I20260812 05:24:38.789992 32144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0waFvT/build/release/bin/kudu
/tmp/dist-test-task0waFvT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786512017613366-32144-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786512017613366-32144-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786512017613366-32144-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786512017613366-32144-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.100.62:33833
--webserver_interface=127.31.100.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.62.keytab
--principal=kudu/127.31.100.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.100.20:43557
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.100.62:33833
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786512017613366-32144-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786512017613366-32144-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786512017613366-32144-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 05:24:38.868618 3884 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 05:24:38.868795 3884 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 05:24:38.868816 3884 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 05:24:38.869890 3884 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 05:24:38.869925 3884 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 05:24:38.869938 3884 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 05:24:38.869949 3884 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 05:24:38.869961 3884 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 05:24:38.871352 3884 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.100.20:43557
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786512017613366-32144-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786512017613366-32144-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.100.62:33833
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.62.keytab
--principal=kudu/127.31.100.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.100.62:33833
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786512017613366-32144-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.100.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3884
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786512017613366-32144-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 04:20:34 UTC on ccycloud.quasar-phqdqp.root.comops.site
build id 4581871
I20260812 05:24:38.871546 3884 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 05:24:38.871753 3884 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 05:24:38.876291 3900 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 05:24:38.876266 3903 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 05:24:38.876281 3899 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 05:24:38.876317 3884 server_base.cc:1034] running on GCE node
I20260812 05:24:38.876701 3884 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 05:24:38.876953 3884 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 05:24:38.878082 3884 hybrid_clock.cc:648] HybridClock initialized: now 1786512278878070 us; error 31 us; skew 500 ppm
Aug 12 05:24:38 dist-test-slave-42z9 krb5kdc[3858](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512278, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.100.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 05:24:38.883111 3884 init.cc:377] Logged in from keytab as kudu/127.31.100.62@KRBTEST.COM (short username kudu)
I20260812 05:24:38.883504 3884 webserver.cc:460] Webserver started at http://127.31.100.62:39261/ using document root <none> and password file <none>
I20260812 05:24:38.883738 3884 fs_manager.cc:359] Metadata directory not provided
I20260812 05:24:38.883785 3884 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 05:24:38.883903 3884 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 05:24:38.884842 3884 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786512017613366-32144-0/minicluster-data/master-0/data/instance:
uuid: "761dc4e50b5b43d085a6dfd869423368"
format_stamp: "Formatted at 2026-08-12 05:24:38 on dist-test-slave-42z9"
server_key: "6c71b2b3c8394e4dec21c7d4affeaab5"
server_key_iv: "897b4da6facb5d5ed4d49ca1b796bf9f"
server_key_version: "encryptionkey@0"
I20260812 05:24:38.885141 3884 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786512017613366-32144-0/minicluster-data/master-0/wal/instance:
uuid: "761dc4e50b5b43d085a6dfd869423368"
format_stamp: "Formatted at 2026-08-12 05:24:38 on dist-test-slave-42z9"
server_key: "6c71b2b3c8394e4dec21c7d4affeaab5"
server_key_iv: "897b4da6facb5d5ed4d49ca1b796bf9f"
server_key_version: "encryptionkey@0"
I20260812 05:24:38.886355 3884 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260812 05:24:38.886981 3915 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 05:24:38.887156 3884 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260812 05:24:38.887240 3884 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786512017613366-32144-0/minicluster-data/master-0/data,/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786512017613366-32144-0/minicluster-data/master-0/wal
uuid: "761dc4e50b5b43d085a6dfd869423368"
format_stamp: "Formatted at 2026-08-12 05:24:38 on dist-test-slave-42z9"
server_key: "6c71b2b3c8394e4dec21c7d4affeaab5"
server_key_iv: "897b4da6facb5d5ed4d49ca1b796bf9f"
server_key_version: "encryptionkey@0"
I20260812 05:24:38.887303 3884 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786512017613366-32144-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786512017613366-32144-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786512017613366-32144-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 05:24:38.905388 3884 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 05:24:38.907002 3884 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 05:24:38.907106 3884 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 05:24:38.910667 3884 rpc_server.cc:307] RPC server started. Bound to: 127.31.100.62:33833
I20260812 05:24:38.910701 4010 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.100.62:33833 every 8 connection(s)
I20260812 05:24:38.910990 3884 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786512017613366-32144-0/minicluster-data/master-0/data/info.pb
I20260812 05:24:38.911749 4011 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 05:24:38.913902 4011 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 761dc4e50b5b43d085a6dfd869423368: Bootstrap starting.
I20260812 05:24:38.914058 32144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0waFvT/build/release/bin/kudu as pid 3884
I20260812 05:24:38.914186 32144 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786512017613366-32144-0/minicluster-data/master-0/wal/instance
I20260812 05:24:38.914378 32144 external_mini_cluster.cc:1428] Setting key 465b9899e2136467c60bedfe85d4809f
I20260812 05:24:38.914585 4011 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 761dc4e50b5b43d085a6dfd869423368: Neither blocks nor log segments found. Creating new log.
I20260812 05:24:38.914914 4011 log.cc:824] T 00000000000000000000000000000000 P 761dc4e50b5b43d085a6dfd869423368: Log is configured to *not* fsync() on all Append() calls
I20260812 05:24:38.915542 4011 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 761dc4e50b5b43d085a6dfd869423368: No bootstrap required, opened a new log
I20260812 05:24:38.917276 4011 raft_consensus.cc:348] T 00000000000000000000000000000000 P 761dc4e50b5b43d085a6dfd869423368 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "761dc4e50b5b43d085a6dfd869423368" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 33833 } }
I20260812 05:24:38.917400 4011 raft_consensus.cc:374] T 00000000000000000000000000000000 P 761dc4e50b5b43d085a6dfd869423368 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 05:24:38.917431 4011 raft_consensus.cc:729] T 00000000000000000000000000000000 P 761dc4e50b5b43d085a6dfd869423368 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 761dc4e50b5b43d085a6dfd869423368, State: Initialized, Role: FOLLOWER
I20260812 05:24:38.917593 4011 consensus_queue.cc:260] T 00000000000000000000000000000000 P 761dc4e50b5b43d085a6dfd869423368 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "761dc4e50b5b43d085a6dfd869423368" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 33833 } }
I20260812 05:24:38.917663 4011 raft_consensus.cc:388] T 00000000000000000000000000000000 P 761dc4e50b5b43d085a6dfd869423368 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 05:24:38.917694 4011 raft_consensus.cc:482] T 00000000000000000000000000000000 P 761dc4e50b5b43d085a6dfd869423368 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 05:24:38.917740 4011 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 761dc4e50b5b43d085a6dfd869423368 [term 0 FOLLOWER]: Advancing to term 1
I20260812 05:24:38.918223 4011 raft_consensus.cc:504] T 00000000000000000000000000000000 P 761dc4e50b5b43d085a6dfd869423368 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "761dc4e50b5b43d085a6dfd869423368" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 33833 } }
I20260812 05:24:38.918319 4011 leader_election.cc:302] T 00000000000000000000000000000000 P 761dc4e50b5b43d085a6dfd869423368 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 761dc4e50b5b43d085a6dfd869423368; no voters:
I20260812 05:24:38.918480 4011 leader_election.cc:288] T 00000000000000000000000000000000 P 761dc4e50b5b43d085a6dfd869423368 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 05:24:38.918498 4022 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 761dc4e50b5b43d085a6dfd869423368 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 05:24:38.918618 4022 raft_consensus.cc:686] T 00000000000000000000000000000000 P 761dc4e50b5b43d085a6dfd869423368 [term 1 LEADER]: Becoming Leader. State: Replica: 761dc4e50b5b43d085a6dfd869423368, State: Running, Role: LEADER
I20260812 05:24:38.918700 4022 consensus_queue.cc:237] T 00000000000000000000000000000000 P 761dc4e50b5b43d085a6dfd869423368 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "761dc4e50b5b43d085a6dfd869423368" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 33833 } }
I20260812 05:24:38.918732 4011 sys_catalog.cc:564] T 00000000000000000000000000000000 P 761dc4e50b5b43d085a6dfd869423368 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 05:24:38.919029 4026 sys_catalog.cc:455] T 00000000000000000000000000000000 P 761dc4e50b5b43d085a6dfd869423368 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 761dc4e50b5b43d085a6dfd869423368. Latest consensus state: current_term: 1 leader_uuid: "761dc4e50b5b43d085a6dfd869423368" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "761dc4e50b5b43d085a6dfd869423368" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 33833 } } }
I20260812 05:24:38.919137 4026 sys_catalog.cc:458] T 00000000000000000000000000000000 P 761dc4e50b5b43d085a6dfd869423368 [sys.catalog]: This master's current role is: LEADER
I20260812 05:24:38.919036 4023 sys_catalog.cc:455] T 00000000000000000000000000000000 P 761dc4e50b5b43d085a6dfd869423368 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "761dc4e50b5b43d085a6dfd869423368" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "761dc4e50b5b43d085a6dfd869423368" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 33833 } } }
I20260812 05:24:38.919219 4023 sys_catalog.cc:458] T 00000000000000000000000000000000 P 761dc4e50b5b43d085a6dfd869423368 [sys.catalog]: This master's current role is: LEADER
I20260812 05:24:38.919409 4034 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 05:24:38.919909 4034 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Aug 12 05:24:38 dist-test-slave-42z9 krb5kdc[3858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512278, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.100.62@KRBTEST.COM
I20260812 05:24:38.921468 4034 catalog_manager.cc:1347] Generated new cluster ID: 3ed3a7ffe40c445eb0d180c05fde471c
I20260812 05:24:38.921521 4034 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 05:24:38.924456 4018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:24:38.914843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49921 (local address 127.31.100.62:33833)
0812 05:24:38.915039 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0812 05:24:38.915048 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 05:24:38.915079 (+ 31us) server_negotiation.cc:407] Connection header received
0812 05:24:38.915488 (+ 409us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:24:38.915495 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:24:38.915537 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:24:38.915657 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:24:38.916273 (+ 616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:24:38.917138 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:24:38.918037 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:24:38.918497 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:24:38.918595 (+ 98us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:24:38.921624 (+ 3029us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 05:24:38.921628 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 05:24:38.921628 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 05:24:38.921640 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0812 05:24:38.923028 (+ 1388us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:24:38.923383 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:24:38.923386 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:24:38.923386 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:24:38.923473 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:24:38.923672 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:24:38.923672 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:24:38.923673 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:24:38.923895 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 05:24:38.924050 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0812 05:24:38.924168 (+ 118us) server_negotiation.cc:299] Negotiation successful
0812 05:24:38.924260 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":84,"threads_started":1}
I20260812 05:24:38.926435 32144 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 05:24:38.930945 4034 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 05:24:38.931589 4034 catalog_manager.cc:1504] Loading token signing keys...
I20260812 05:24:38.933341 4018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:24:38.927670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50031 (local address 127.31.100.62:33833)
0812 05:24:38.927702 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0812 05:24:38.927704 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 05:24:38.927808 (+ 104us) server_negotiation.cc:407] Connection header received
0812 05:24:38.927868 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:24:38.927869 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:24:38.927877 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:24:38.927921 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:24:38.928402 (+ 481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:24:38.928852 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:24:38.929543 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:24:38.929764 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:24:38.929817 (+ 53us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:24:38.931116 (+ 1299us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 05:24:38.931117 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 05:24:38.931118 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 05:24:38.931124 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0812 05:24:38.932307 (+ 1183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:24:38.932647 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:24:38.932648 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:24:38.932648 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:24:38.932700 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:24:38.932889 (+ 189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:24:38.932890 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:24:38.932890 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:24:38.932986 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 05:24:38.933067 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0812 05:24:38.933209 (+ 142us) server_negotiation.cc:299] Negotiation successful
0812 05:24:38.933264 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260812 05:24:38.936173 4034 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 761dc4e50b5b43d085a6dfd869423368: Generated new TSK 0
I20260812 05:24:38.936326 4034 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 05:24:38.940093 32144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0waFvT/build/release/bin/kudu with pid 3884
W20260812 05:24:38.956063 3855 connection.cc:535] client connection to 127.31.100.62:33833 recv error: Network error: failed to read from TLS socket (remote: 127.31.100.62:33833): Connection reset by peer (error 104)
2026-08-12T05:24:38Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1420 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786512017613366-32144-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 05:24:38 dist-test-slave-42z9 krb5kdc[4065](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 05:24:38 dist-test-slave-42z9 krb5kdc[4065](info): set up 2 sockets
Aug 12 05:24:38 dist-test-slave-42z9 krb5kdc[4065](info): commencing operation
krb5kdc: starting...
W20260812 05:24:40.104393 32144 mini_kdc.cc:121] Time spent starting KDC: real 1.135s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786512017613366-32144-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786512017613366-32144-0/krb5kdc/test-user.keytab.
Aug 12 05:24:40 dist-test-slave-42z9 krb5kdc[4065](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512280, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T05:24:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T05:24:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.100.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.100.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.100.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.62.keytab.
Entry for principal kudu/127.31.100.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.62.keytab.
I20260812 05:24:40.223590 32144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0waFvT/build/release/bin/kudu
/tmp/dist-test-task0waFvT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786512017613366-32144-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786512017613366-32144-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786512017613366-32144-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786512017613366-32144-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.100.62:35155
--webserver_interface=127.31.100.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.62.keytab
--principal=kudu/127.31.100.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.100.20:38915
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.100.62:35155
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786512017613366-32144-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786512017613366-32144-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786512017613366-32144-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 05:24:40.303933 4088 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 05:24:40.304136 4088 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 05:24:40.304157 4088 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 05:24:40.305280 4088 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 05:24:40.305321 4088 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 05:24:40.305334 4088 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 05:24:40.305357 4088 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 05:24:40.305380 4088 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 05:24:40.306882 4088 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.100.20:38915
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786512017613366-32144-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786512017613366-32144-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.100.62:35155
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.62.keytab
--principal=kudu/127.31.100.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.100.62:35155
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786512017613366-32144-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.100.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4088
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786512017613366-32144-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 04:20:34 UTC on ccycloud.quasar-phqdqp.root.comops.site
build id 4581871
I20260812 05:24:40.307126 4088 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 05:24:40.307340 4088 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 05:24:40.311973 4097 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 05:24:40.311975 4095 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 05:24:40.312000 4100 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 05:24:40.312003 4088 server_base.cc:1034] running on GCE node
I20260812 05:24:40.312418 4088 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 05:24:40.312695 4088 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 05:24:40.313830 4088 hybrid_clock.cc:648] HybridClock initialized: now 1786512280313816 us; error 27 us; skew 500 ppm
Aug 12 05:24:40 dist-test-slave-42z9 krb5kdc[4065](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512280, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.100.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 05:24:40.316150 4088 init.cc:377] Logged in from keytab as kudu/127.31.100.62@KRBTEST.COM (short username kudu)
I20260812 05:24:40.316536 4088 webserver.cc:460] Webserver started at http://127.31.100.62:36233/ using document root <none> and password file <none>
I20260812 05:24:40.316751 4088 fs_manager.cc:359] Metadata directory not provided
I20260812 05:24:40.316798 4088 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 05:24:40.316926 4088 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 05:24:40.317963 4088 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786512017613366-32144-0/minicluster-data/master-0/data/instance:
uuid: "33b829f0eeb14b22896be4b41b7fc977"
format_stamp: "Formatted at 2026-08-12 05:24:40 on dist-test-slave-42z9"
server_key: "973628f7627fdab1f9798507f7ae4c6c"
server_key_iv: "854f7217205893ffac367696e7895f0b"
server_key_version: "encryptionkey@0"
I20260812 05:24:40.318287 4088 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786512017613366-32144-0/minicluster-data/master-0/wal/instance:
uuid: "33b829f0eeb14b22896be4b41b7fc977"
format_stamp: "Formatted at 2026-08-12 05:24:40 on dist-test-slave-42z9"
server_key: "973628f7627fdab1f9798507f7ae4c6c"
server_key_iv: "854f7217205893ffac367696e7895f0b"
server_key_version: "encryptionkey@0"
I20260812 05:24:40.319484 4088 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260812 05:24:40.320109 4111 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 05:24:40.320251 4088 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260812 05:24:40.320338 4088 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786512017613366-32144-0/minicluster-data/master-0/data,/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786512017613366-32144-0/minicluster-data/master-0/wal
uuid: "33b829f0eeb14b22896be4b41b7fc977"
format_stamp: "Formatted at 2026-08-12 05:24:40 on dist-test-slave-42z9"
server_key: "973628f7627fdab1f9798507f7ae4c6c"
server_key_iv: "854f7217205893ffac367696e7895f0b"
server_key_version: "encryptionkey@0"
I20260812 05:24:40.320394 4088 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786512017613366-32144-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786512017613366-32144-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786512017613366-32144-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 05:24:40.336935 4088 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 05:24:40.339170 4088 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 05:24:40.339313 4088 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 05:24:40.343228 4088 rpc_server.cc:307] RPC server started. Bound to: 127.31.100.62:35155
I20260812 05:24:40.343266 4198 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.100.62:35155 every 8 connection(s)
I20260812 05:24:40.343588 4088 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786512017613366-32144-0/minicluster-data/master-0/data/info.pb
I20260812 05:24:40.344460 4199 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 05:24:40.346729 4199 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 33b829f0eeb14b22896be4b41b7fc977: Bootstrap starting.
I20260812 05:24:40.347311 4199 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 33b829f0eeb14b22896be4b41b7fc977: Neither blocks nor log segments found. Creating new log.
I20260812 05:24:40.347561 4199 log.cc:824] T 00000000000000000000000000000000 P 33b829f0eeb14b22896be4b41b7fc977: Log is configured to *not* fsync() on all Append() calls
I20260812 05:24:40.348044 32144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0waFvT/build/release/bin/kudu as pid 4088
I20260812 05:24:40.348168 32144 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786512017613366-32144-0/minicluster-data/master-0/wal/instance
I20260812 05:24:40.348287 4199 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 33b829f0eeb14b22896be4b41b7fc977: No bootstrap required, opened a new log
I20260812 05:24:40.348371 32144 external_mini_cluster.cc:1428] Setting key bd1c02dd4855f09bd353af2ddd846646
I20260812 05:24:40.350716 4199 raft_consensus.cc:348] T 00000000000000000000000000000000 P 33b829f0eeb14b22896be4b41b7fc977 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33b829f0eeb14b22896be4b41b7fc977" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 35155 } }
I20260812 05:24:40.350860 4199 raft_consensus.cc:374] T 00000000000000000000000000000000 P 33b829f0eeb14b22896be4b41b7fc977 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 05:24:40.350903 4199 raft_consensus.cc:729] T 00000000000000000000000000000000 P 33b829f0eeb14b22896be4b41b7fc977 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33b829f0eeb14b22896be4b41b7fc977, State: Initialized, Role: FOLLOWER
I20260812 05:24:40.351029 4199 consensus_queue.cc:260] T 00000000000000000000000000000000 P 33b829f0eeb14b22896be4b41b7fc977 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33b829f0eeb14b22896be4b41b7fc977" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 35155 } }
I20260812 05:24:40.351106 4199 raft_consensus.cc:388] T 00000000000000000000000000000000 P 33b829f0eeb14b22896be4b41b7fc977 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 05:24:40.351140 4199 raft_consensus.cc:482] T 00000000000000000000000000000000 P 33b829f0eeb14b22896be4b41b7fc977 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 05:24:40.351184 4199 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 33b829f0eeb14b22896be4b41b7fc977 [term 0 FOLLOWER]: Advancing to term 1
I20260812 05:24:40.351833 4199 raft_consensus.cc:504] T 00000000000000000000000000000000 P 33b829f0eeb14b22896be4b41b7fc977 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33b829f0eeb14b22896be4b41b7fc977" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 35155 } }
I20260812 05:24:40.351933 4199 leader_election.cc:302] T 00000000000000000000000000000000 P 33b829f0eeb14b22896be4b41b7fc977 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 33b829f0eeb14b22896be4b41b7fc977; no voters:
I20260812 05:24:40.352084 4199 leader_election.cc:288] T 00000000000000000000000000000000 P 33b829f0eeb14b22896be4b41b7fc977 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 05:24:40.352116 4210 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 33b829f0eeb14b22896be4b41b7fc977 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 05:24:40.352219 4210 raft_consensus.cc:686] T 00000000000000000000000000000000 P 33b829f0eeb14b22896be4b41b7fc977 [term 1 LEADER]: Becoming Leader. State: Replica: 33b829f0eeb14b22896be4b41b7fc977, State: Running, Role: LEADER
I20260812 05:24:40.352341 4199 sys_catalog.cc:564] T 00000000000000000000000000000000 P 33b829f0eeb14b22896be4b41b7fc977 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 05:24:40.352360 4210 consensus_queue.cc:237] T 00000000000000000000000000000000 P 33b829f0eeb14b22896be4b41b7fc977 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33b829f0eeb14b22896be4b41b7fc977" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 35155 } }
I20260812 05:24:40.352764 4210 sys_catalog.cc:455] T 00000000000000000000000000000000 P 33b829f0eeb14b22896be4b41b7fc977 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 33b829f0eeb14b22896be4b41b7fc977. Latest consensus state: current_term: 1 leader_uuid: "33b829f0eeb14b22896be4b41b7fc977" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33b829f0eeb14b22896be4b41b7fc977" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 35155 } } }
I20260812 05:24:40.352895 4210 sys_catalog.cc:458] T 00000000000000000000000000000000 P 33b829f0eeb14b22896be4b41b7fc977 [sys.catalog]: This master's current role is: LEADER
I20260812 05:24:40.352828 4211 sys_catalog.cc:455] T 00000000000000000000000000000000 P 33b829f0eeb14b22896be4b41b7fc977 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "33b829f0eeb14b22896be4b41b7fc977" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33b829f0eeb14b22896be4b41b7fc977" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 35155 } } }
I20260812 05:24:40.353132 4211 sys_catalog.cc:458] T 00000000000000000000000000000000 P 33b829f0eeb14b22896be4b41b7fc977 [sys.catalog]: This master's current role is: LEADER
I20260812 05:24:40.353168 4223 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 05:24:40.353826 4207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:24:40.348880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43723 (local address 127.31.100.62:35155)
0812 05:24:40.349066 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0812 05:24:40.349073 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 05:24:40.349099 (+ 26us) server_negotiation.cc:407] Connection header received
0812 05:24:40.349540 (+ 441us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:24:40.349547 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:24:40.349596 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:24:40.349710 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:24:40.349920 (+ 210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:24:40.350757 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:24:40.351797 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:24:40.352300 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:24:40.352408 (+ 108us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:24:40.353050 (+ 642us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 05:24:40.353053 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 05:24:40.353054 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 05:24:40.353258 (+ 204us) server_negotiation.cc:922] calling sasl_server_start()
0812 05:24:40.353334 (+ 76us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 05:24:40.353350 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 05:24:40.353425 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0812 05:24:40.353543 (+ 118us) server_negotiation.cc:299] Negotiation successful
0812 05:24:40.353572 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":76,"threads_started":1}
I20260812 05:24:40.354063 4223 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 05:24:40.354959 32144 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 05:24:40.355623 4223 catalog_manager.cc:1347] Generated new cluster ID: 006a8b4c2a194080948e1cf10d1185a8
I20260812 05:24:40.355677 4223 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 05:24:40.360814 4207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:24:40.356377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41965 (local address 127.31.100.62:35155)
0812 05:24:40.356406 (+ 29us) server_negotiation.cc:206] Beginning negotiation
0812 05:24:40.356408 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 05:24:40.357646 (+ 1238us) server_negotiation.cc:407] Connection header received
0812 05:24:40.357757 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:24:40.357758 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:24:40.357769 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:24:40.357829 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:24:40.357957 (+ 128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:24:40.358596 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:24:40.359575 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:24:40.359912 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:24:40.359987 (+ 75us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:24:40.360582 (+ 595us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 05:24:40.360583 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 05:24:40.360584 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 05:24:40.360588 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0812 05:24:40.360611 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 05:24:40.360616 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 05:24:40.360679 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0812 05:24:40.360731 (+ 52us) server_negotiation.cc:299] Negotiation successful
0812 05:24:40.360741 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14}
I20260812 05:24:40.380746 4223 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 05:24:40.381488 4223 catalog_manager.cc:1504] Loading token signing keys...
I20260812 05:24:40.389853 4223 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 33b829f0eeb14b22896be4b41b7fc977: Generated new TSK 0
I20260812 05:24:40.390040 4223 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 05:24:40.397128 32144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0waFvT/build/release/bin/kudu with pid 4088
W20260812 05:24:40.414718 4063 connection.cc:535] client connection to 127.31.100.62:35155 recv error: Network error: failed to read from TLS socket (remote: 127.31.100.62:35155): Connection reset by peer (error 104)
2026-08-12T05:24:40Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1458 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786512017613366-32144-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 05:24:40 dist-test-slave-42z9 krb5kdc[4254](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 05:24:40 dist-test-slave-42z9 krb5kdc[4254](info): set up 2 sockets
Aug 12 05:24:40 dist-test-slave-42z9 krb5kdc[4254](info): commencing operation
krb5kdc: starting...
W20260812 05:24:41.577733 32144 mini_kdc.cc:121] Time spent starting KDC: real 1.150s user 0.004s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786512017613366-32144-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786512017613366-32144-0/krb5kdc/test-user.keytab.
Aug 12 05:24:41 dist-test-slave-42z9 krb5kdc[4254](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512281, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T05:24:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T05:24:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.100.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.100.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.100.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.62.keytab.
Entry for principal kudu/127.31.100.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.62.keytab.
I20260812 05:24:41.688578 32144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0waFvT/build/release/bin/kudu
/tmp/dist-test-task0waFvT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786512017613366-32144-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786512017613366-32144-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786512017613366-32144-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786512017613366-32144-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.100.62:46017
--webserver_interface=127.31.100.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.62.keytab
--principal=kudu/127.31.100.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.100.20:33197
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.100.62:46017
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786512017613366-32144-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786512017613366-32144-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786512017613366-32144-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 05:24:41.767796 4272 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 05:24:41.767980 4272 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 05:24:41.768013 4272 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 05:24:41.769094 4272 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 05:24:41.769135 4272 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 05:24:41.769155 4272 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 05:24:41.769170 4272 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 05:24:41.769201 4272 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 05:24:41.770846 4272 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.100.20:33197
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786512017613366-32144-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786512017613366-32144-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.100.62:46017
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.62.keytab
--principal=kudu/127.31.100.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.100.62:46017
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786512017613366-32144-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.100.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4272
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786512017613366-32144-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 04:20:34 UTC on ccycloud.quasar-phqdqp.root.comops.site
build id 4581871
I20260812 05:24:41.771080 4272 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 05:24:41.771281 4272 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 05:24:41.775806 4280 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 05:24:41.775815 4279 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 05:24:41.775813 4289 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 05:24:41.775843 4272 server_base.cc:1034] running on GCE node
I20260812 05:24:41.776230 4272 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 05:24:41.776454 4272 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 05:24:41.777581 4272 hybrid_clock.cc:648] HybridClock initialized: now 1786512281777565 us; error 31 us; skew 500 ppm
Aug 12 05:24:41 dist-test-slave-42z9 krb5kdc[4254](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512281, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.100.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 05:24:41.779898 4272 init.cc:377] Logged in from keytab as kudu/127.31.100.62@KRBTEST.COM (short username kudu)
I20260812 05:24:41.780263 4272 webserver.cc:460] Webserver started at http://127.31.100.62:36597/ using document root <none> and password file <none>
I20260812 05:24:41.780448 4272 fs_manager.cc:359] Metadata directory not provided
I20260812 05:24:41.780486 4272 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 05:24:41.780570 4272 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 05:24:41.781523 4272 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786512017613366-32144-0/minicluster-data/master-0/data/instance:
uuid: "64fb7bb696564a6bb6df997ebb343bc4"
format_stamp: "Formatted at 2026-08-12 05:24:41 on dist-test-slave-42z9"
server_key: "162ee4ca9d8cd80fdcb81e14109317a7"
server_key_iv: "3e51d5e6dacb0c546d089ed3103ab01e"
server_key_version: "encryptionkey@0"
I20260812 05:24:41.781813 4272 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786512017613366-32144-0/minicluster-data/master-0/wal/instance:
uuid: "64fb7bb696564a6bb6df997ebb343bc4"
format_stamp: "Formatted at 2026-08-12 05:24:41 on dist-test-slave-42z9"
server_key: "162ee4ca9d8cd80fdcb81e14109317a7"
server_key_iv: "3e51d5e6dacb0c546d089ed3103ab01e"
server_key_version: "encryptionkey@0"
I20260812 05:24:41.782949 4272 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260812 05:24:41.783553 4298 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 05:24:41.783699 4272 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260812 05:24:41.783782 4272 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786512017613366-32144-0/minicluster-data/master-0/data,/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786512017613366-32144-0/minicluster-data/master-0/wal
uuid: "64fb7bb696564a6bb6df997ebb343bc4"
format_stamp: "Formatted at 2026-08-12 05:24:41 on dist-test-slave-42z9"
server_key: "162ee4ca9d8cd80fdcb81e14109317a7"
server_key_iv: "3e51d5e6dacb0c546d089ed3103ab01e"
server_key_version: "encryptionkey@0"
I20260812 05:24:41.783840 4272 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786512017613366-32144-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786512017613366-32144-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786512017613366-32144-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 05:24:41.786967 4272 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 05:24:41.788959 4272 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 05:24:41.789081 4272 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 05:24:41.792804 4272 rpc_server.cc:307] RPC server started. Bound to: 127.31.100.62:46017
I20260812 05:24:41.792837 4391 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.100.62:46017 every 8 connection(s)
I20260812 05:24:41.793130 4272 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786512017613366-32144-0/minicluster-data/master-0/data/info.pb
I20260812 05:24:41.794055 4392 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 05:24:41.796123 4392 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 64fb7bb696564a6bb6df997ebb343bc4: Bootstrap starting.
I20260812 05:24:41.796656 4392 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 64fb7bb696564a6bb6df997ebb343bc4: Neither blocks nor log segments found. Creating new log.
I20260812 05:24:41.796902 4392 log.cc:824] T 00000000000000000000000000000000 P 64fb7bb696564a6bb6df997ebb343bc4: Log is configured to *not* fsync() on all Append() calls
I20260812 05:24:41.797544 4392 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 64fb7bb696564a6bb6df997ebb343bc4: No bootstrap required, opened a new log
I20260812 05:24:41.799103 4392 raft_consensus.cc:348] T 00000000000000000000000000000000 P 64fb7bb696564a6bb6df997ebb343bc4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64fb7bb696564a6bb6df997ebb343bc4" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 46017 } }
I20260812 05:24:41.799232 4392 raft_consensus.cc:374] T 00000000000000000000000000000000 P 64fb7bb696564a6bb6df997ebb343bc4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 05:24:41.799264 4392 raft_consensus.cc:729] T 00000000000000000000000000000000 P 64fb7bb696564a6bb6df997ebb343bc4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64fb7bb696564a6bb6df997ebb343bc4, State: Initialized, Role: FOLLOWER
I20260812 05:24:41.799410 4392 consensus_queue.cc:260] T 00000000000000000000000000000000 P 64fb7bb696564a6bb6df997ebb343bc4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64fb7bb696564a6bb6df997ebb343bc4" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 46017 } }
I20260812 05:24:41.799489 4392 raft_consensus.cc:388] T 00000000000000000000000000000000 P 64fb7bb696564a6bb6df997ebb343bc4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 05:24:41.799525 4392 raft_consensus.cc:482] T 00000000000000000000000000000000 P 64fb7bb696564a6bb6df997ebb343bc4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 05:24:41.799574 4392 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 64fb7bb696564a6bb6df997ebb343bc4 [term 0 FOLLOWER]: Advancing to term 1
I20260812 05:24:41.800446 4392 raft_consensus.cc:504] T 00000000000000000000000000000000 P 64fb7bb696564a6bb6df997ebb343bc4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64fb7bb696564a6bb6df997ebb343bc4" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 46017 } }
I20260812 05:24:41.800536 4392 leader_election.cc:302] T 00000000000000000000000000000000 P 64fb7bb696564a6bb6df997ebb343bc4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 64fb7bb696564a6bb6df997ebb343bc4; no voters:
I20260812 05:24:41.800676 4392 leader_election.cc:288] T 00000000000000000000000000000000 P 64fb7bb696564a6bb6df997ebb343bc4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 05:24:41.800729 4396 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 64fb7bb696564a6bb6df997ebb343bc4 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 05:24:41.800900 4396 raft_consensus.cc:686] T 00000000000000000000000000000000 P 64fb7bb696564a6bb6df997ebb343bc4 [term 1 LEADER]: Becoming Leader. State: Replica: 64fb7bb696564a6bb6df997ebb343bc4, State: Running, Role: LEADER
I20260812 05:24:41.800899 4392 sys_catalog.cc:564] T 00000000000000000000000000000000 P 64fb7bb696564a6bb6df997ebb343bc4 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 05:24:41.801002 4396 consensus_queue.cc:237] T 00000000000000000000000000000000 P 64fb7bb696564a6bb6df997ebb343bc4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64fb7bb696564a6bb6df997ebb343bc4" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 46017 } }
I20260812 05:24:41.801378 4398 sys_catalog.cc:455] T 00000000000000000000000000000000 P 64fb7bb696564a6bb6df997ebb343bc4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 64fb7bb696564a6bb6df997ebb343bc4. Latest consensus state: current_term: 1 leader_uuid: "64fb7bb696564a6bb6df997ebb343bc4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64fb7bb696564a6bb6df997ebb343bc4" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 46017 } } }
I20260812 05:24:41.801393 4397 sys_catalog.cc:455] T 00000000000000000000000000000000 P 64fb7bb696564a6bb6df997ebb343bc4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "64fb7bb696564a6bb6df997ebb343bc4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64fb7bb696564a6bb6df997ebb343bc4" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 46017 } } }
I20260812 05:24:41.801525 4398 sys_catalog.cc:458] T 00000000000000000000000000000000 P 64fb7bb696564a6bb6df997ebb343bc4 [sys.catalog]: This master's current role is: LEADER
I20260812 05:24:41.801535 4397 sys_catalog.cc:458] T 00000000000000000000000000000000 P 64fb7bb696564a6bb6df997ebb343bc4 [sys.catalog]: This master's current role is: LEADER
I20260812 05:24:41.801819 4407 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 05:24:41.802320 4407 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 05:24:41.802578 32144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0waFvT/build/release/bin/kudu as pid 4272
I20260812 05:24:41.802712 32144 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786512017613366-32144-0/minicluster-data/master-0/wal/instance
I20260812 05:24:41.802924 32144 external_mini_cluster.cc:1428] Setting key 3c04cee0b7a6f225f692343e3ab93d8d
I20260812 05:24:41.804981 4427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:24:41.803581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48607 (local address 127.31.100.62:46017)
0812 05:24:41.803706 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0812 05:24:41.803714 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 05:24:41.803745 (+ 31us) server_negotiation.cc:407] Connection header received
0812 05:24:41.803850 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:24:41.803857 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:24:41.803896 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:24:41.804027 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:24:41.804605 (+ 578us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 05:24:41.804608 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 05:24:41.804609 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 05:24:41.804613 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0812 05:24:41.804649 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 05:24:41.804662 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 05:24:41.804732 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0812 05:24:41.804768 (+ 36us) server_negotiation.cc:299] Negotiation successful
0812 05:24:41.804808 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81,"thread_start_us":52,"threads_started":1}
I20260812 05:24:41.805863 32144 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 05:24:41.809172 4427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:24:41.807474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54431 (local address 127.31.100.62:46017)
0812 05:24:41.808287 (+ 813us) server_negotiation.cc:206] Beginning negotiation
0812 05:24:41.808289 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 05:24:41.808423 (+ 134us) server_negotiation.cc:407] Connection header received
0812 05:24:41.808489 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:24:41.808490 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:24:41.808495 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:24:41.808539 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:24:41.808990 (+ 451us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 05:24:41.808991 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 05:24:41.808991 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 05:24:41.808994 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0812 05:24:41.809007 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 05:24:41.809010 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 05:24:41.809051 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0812 05:24:41.809116 (+ 65us) server_negotiation.cc:299] Negotiation successful
0812 05:24:41.809126 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":790}
I20260812 05:24:41.813795 4407 catalog_manager.cc:1347] Generated new cluster ID: 32a2eb2d0c1a43afad2c1d72366f260b
I20260812 05:24:41.813843 4407 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 05:24:41.825054 4407 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 05:24:41.825191 4407 catalog_manager.cc:1504] Loading token signing keys...
I20260812 05:24:41.829581 4407 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 64fb7bb696564a6bb6df997ebb343bc4: Generated new TSK 0
I20260812 05:24:41.829727 4407 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 05:24:41.835855 32144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0waFvT/build/release/bin/kudu with pid 4272
2026-08-12T05:24:41Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1432 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786512017613366-32144-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 05:24:41 dist-test-slave-42z9 krb5kdc[4439](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 05:24:41 dist-test-slave-42z9 krb5kdc[4439](info): set up 2 sockets
Aug 12 05:24:41 dist-test-slave-42z9 krb5kdc[4439](info): commencing operation
krb5kdc: starting...
W20260812 05:24:42.993551 32144 mini_kdc.cc:121] Time spent starting KDC: real 1.133s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786512017613366-32144-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786512017613366-32144-0/krb5kdc/test-user.keytab.
Aug 12 05:24:43 dist-test-slave-42z9 krb5kdc[4439](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512283, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T05:24:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T05:24:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.100.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.100.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.100.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.62.keytab.
Entry for principal kudu/127.31.100.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.62.keytab.
I20260812 05:24:43.106084 32144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0waFvT/build/release/bin/kudu
/tmp/dist-test-task0waFvT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786512017613366-32144-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786512017613366-32144-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786512017613366-32144-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786512017613366-32144-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.100.62:33669
--webserver_interface=127.31.100.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.62.keytab
--principal=kudu/127.31.100.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.100.20:40183
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.100.62:33669
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786512017613366-32144-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786512017613366-32144-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786512017613366-32144-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 05:24:43.189958 4459 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 05:24:43.190166 4459 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 05:24:43.190196 4459 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 05:24:43.191373 4459 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 05:24:43.191417 4459 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 05:24:43.191437 4459 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 05:24:43.191452 4459 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 05:24:43.191468 4459 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 05:24:43.192927 4459 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.100.20:40183
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786512017613366-32144-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786512017613366-32144-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.100.62:33669
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.62.keytab
--principal=kudu/127.31.100.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.100.62:33669
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786512017613366-32144-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.100.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4459
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786512017613366-32144-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 04:20:34 UTC on ccycloud.quasar-phqdqp.root.comops.site
build id 4581871
I20260812 05:24:43.193181 4459 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 05:24:43.193383 4459 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 05:24:43.197971 4476 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 05:24:43.198045 4469 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 05:24:43.198004 4465 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 05:24:43.198288 4459 server_base.cc:1034] running on GCE node
I20260812 05:24:43.198534 4459 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 05:24:43.198808 4459 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 05:24:43.199956 4459 hybrid_clock.cc:648] HybridClock initialized: now 1786512283199937 us; error 33 us; skew 500 ppm
Aug 12 05:24:43 dist-test-slave-42z9 krb5kdc[4439](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512283, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.100.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 05:24:43.202276 4459 init.cc:377] Logged in from keytab as kudu/127.31.100.62@KRBTEST.COM (short username kudu)
I20260812 05:24:43.202744 4459 webserver.cc:460] Webserver started at http://127.31.100.62:34941/ using document root <none> and password file <none>
I20260812 05:24:43.203020 4459 fs_manager.cc:359] Metadata directory not provided
I20260812 05:24:43.203085 4459 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 05:24:43.203212 4459 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 05:24:43.204224 4459 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786512017613366-32144-0/minicluster-data/master-0/data/instance:
uuid: "52df487fc568476081527b762b9ac395"
format_stamp: "Formatted at 2026-08-12 05:24:43 on dist-test-slave-42z9"
server_key: "94e0be5a1e611b19c722a613b00e101e"
server_key_iv: "a5711fcae21a5cf88fd945b19b8fe74f"
server_key_version: "encryptionkey@0"
I20260812 05:24:43.204552 4459 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786512017613366-32144-0/minicluster-data/master-0/wal/instance:
uuid: "52df487fc568476081527b762b9ac395"
format_stamp: "Formatted at 2026-08-12 05:24:43 on dist-test-slave-42z9"
server_key: "94e0be5a1e611b19c722a613b00e101e"
server_key_iv: "a5711fcae21a5cf88fd945b19b8fe74f"
server_key_version: "encryptionkey@0"
I20260812 05:24:43.205799 4459 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260812 05:24:43.206450 4486 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 05:24:43.206602 4459 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260812 05:24:43.206689 4459 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786512017613366-32144-0/minicluster-data/master-0/data,/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786512017613366-32144-0/minicluster-data/master-0/wal
uuid: "52df487fc568476081527b762b9ac395"
format_stamp: "Formatted at 2026-08-12 05:24:43 on dist-test-slave-42z9"
server_key: "94e0be5a1e611b19c722a613b00e101e"
server_key_iv: "a5711fcae21a5cf88fd945b19b8fe74f"
server_key_version: "encryptionkey@0"
I20260812 05:24:43.206746 4459 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786512017613366-32144-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786512017613366-32144-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786512017613366-32144-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 05:24:43.234432 4459 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 05:24:43.236083 4459 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 05:24:43.236217 4459 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 05:24:43.240151 4459 rpc_server.cc:307] RPC server started. Bound to: 127.31.100.62:33669
I20260812 05:24:43.240186 4561 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.100.62:33669 every 8 connection(s)
I20260812 05:24:43.240528 4459 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786512017613366-32144-0/minicluster-data/master-0/data/info.pb
I20260812 05:24:43.241415 4562 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 05:24:43.243719 4562 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 52df487fc568476081527b762b9ac395: Bootstrap starting.
I20260812 05:24:43.244279 4562 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 52df487fc568476081527b762b9ac395: Neither blocks nor log segments found. Creating new log.
I20260812 05:24:43.244537 4562 log.cc:824] T 00000000000000000000000000000000 P 52df487fc568476081527b762b9ac395: Log is configured to *not* fsync() on all Append() calls
I20260812 05:24:43.245175 4562 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 52df487fc568476081527b762b9ac395: No bootstrap required, opened a new log
I20260812 05:24:43.246773 4562 raft_consensus.cc:348] T 00000000000000000000000000000000 P 52df487fc568476081527b762b9ac395 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52df487fc568476081527b762b9ac395" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 33669 } }
I20260812 05:24:43.246905 4562 raft_consensus.cc:374] T 00000000000000000000000000000000 P 52df487fc568476081527b762b9ac395 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 05:24:43.246927 4562 raft_consensus.cc:729] T 00000000000000000000000000000000 P 52df487fc568476081527b762b9ac395 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52df487fc568476081527b762b9ac395, State: Initialized, Role: FOLLOWER
I20260812 05:24:43.247053 4562 consensus_queue.cc:260] T 00000000000000000000000000000000 P 52df487fc568476081527b762b9ac395 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52df487fc568476081527b762b9ac395" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 33669 } }
I20260812 05:24:43.247112 4562 raft_consensus.cc:388] T 00000000000000000000000000000000 P 52df487fc568476081527b762b9ac395 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 05:24:43.247135 4562 raft_consensus.cc:482] T 00000000000000000000000000000000 P 52df487fc568476081527b762b9ac395 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 05:24:43.247182 4562 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 52df487fc568476081527b762b9ac395 [term 0 FOLLOWER]: Advancing to term 1
I20260812 05:24:43.247963 4562 raft_consensus.cc:504] T 00000000000000000000000000000000 P 52df487fc568476081527b762b9ac395 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52df487fc568476081527b762b9ac395" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 33669 } }
I20260812 05:24:43.248055 4562 leader_election.cc:302] T 00000000000000000000000000000000 P 52df487fc568476081527b762b9ac395 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 52df487fc568476081527b762b9ac395; no voters:
I20260812 05:24:43.248205 4562 leader_election.cc:288] T 00000000000000000000000000000000 P 52df487fc568476081527b762b9ac395 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 05:24:43.248243 4567 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 52df487fc568476081527b762b9ac395 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 05:24:43.248428 4567 raft_consensus.cc:686] T 00000000000000000000000000000000 P 52df487fc568476081527b762b9ac395 [term 1 LEADER]: Becoming Leader. State: Replica: 52df487fc568476081527b762b9ac395, State: Running, Role: LEADER
I20260812 05:24:43.248437 4562 sys_catalog.cc:564] T 00000000000000000000000000000000 P 52df487fc568476081527b762b9ac395 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 05:24:43.248593 4567 consensus_queue.cc:237] T 00000000000000000000000000000000 P 52df487fc568476081527b762b9ac395 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52df487fc568476081527b762b9ac395" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 33669 } }
I20260812 05:24:43.248967 4569 sys_catalog.cc:455] T 00000000000000000000000000000000 P 52df487fc568476081527b762b9ac395 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "52df487fc568476081527b762b9ac395" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52df487fc568476081527b762b9ac395" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 33669 } } }
I20260812 05:24:43.249053 4569 sys_catalog.cc:458] T 00000000000000000000000000000000 P 52df487fc568476081527b762b9ac395 [sys.catalog]: This master's current role is: LEADER
I20260812 05:24:43.249200 4568 sys_catalog.cc:455] T 00000000000000000000000000000000 P 52df487fc568476081527b762b9ac395 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 52df487fc568476081527b762b9ac395. Latest consensus state: current_term: 1 leader_uuid: "52df487fc568476081527b762b9ac395" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52df487fc568476081527b762b9ac395" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 33669 } } }
I20260812 05:24:43.249262 4568 sys_catalog.cc:458] T 00000000000000000000000000000000 P 52df487fc568476081527b762b9ac395 [sys.catalog]: This master's current role is: LEADER
I20260812 05:24:43.249533 4578 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 05:24:43.250054 4578 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 05:24:43.250552 32144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0waFvT/build/release/bin/kudu as pid 4459
I20260812 05:24:43.250666 32144 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786512017613366-32144-0/minicluster-data/master-0/wal/instance
I20260812 05:24:43.250839 32144 external_mini_cluster.cc:1428] Setting key beca9470344b3133ed088c399a243a34
I20260812 05:24:43.251878 4578 catalog_manager.cc:1347] Generated new cluster ID: 4997b8b41ed14b2ab52f5b4834d37e3a
I20260812 05:24:43.251932 4578 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 12 05:24:43 dist-test-slave-42z9 krb5kdc[4439](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512283, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.100.62@KRBTEST.COM
I20260812 05:24:43.264062 4596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:24:43.251290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46089 (local address 127.31.100.62:33669)
0812 05:24:43.251572 (+ 282us) server_negotiation.cc:206] Beginning negotiation
0812 05:24:43.251581 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 05:24:43.251616 (+ 35us) server_negotiation.cc:407] Connection header received
0812 05:24:43.251729 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:24:43.251736 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:24:43.251799 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:24:43.253595 (+ 1796us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:24:43.254358 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:24:43.256143 (+ 1785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:24:43.257213 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:24:43.257857 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:24:43.257957 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:24:43.260673 (+ 2716us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 05:24:43.260677 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 05:24:43.260677 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 05:24:43.260689 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0812 05:24:43.262521 (+ 1832us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:24:43.262869 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:24:43.262872 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:24:43.262872 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:24:43.262983 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:24:43.263147 (+ 164us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:24:43.263148 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:24:43.263148 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:24:43.263439 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 05:24:43.263568 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0812 05:24:43.263669 (+ 101us) server_negotiation.cc:299] Negotiation successful
0812 05:24:43.263811 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":241,"thread_start_us":75,"threads_started":1}
I20260812 05:24:43.265014 32144 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 05:24:43.273342 4596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:24:43.266396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37159 (local address 127.31.100.62:33669)
0812 05:24:43.266429 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0812 05:24:43.266431 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 05:24:43.266532 (+ 101us) server_negotiation.cc:407] Connection header received
0812 05:24:43.266593 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:24:43.266594 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:24:43.266602 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:24:43.266644 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:24:43.267196 (+ 552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:24:43.267904 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:24:43.268865 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:24:43.269149 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:24:43.269199 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:24:43.270417 (+ 1218us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 05:24:43.270418 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 05:24:43.270419 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 05:24:43.270425 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0812 05:24:43.272331 (+ 1906us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:24:43.272691 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:24:43.272692 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:24:43.272692 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:24:43.272742 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:24:43.272913 (+ 171us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:24:43.272913 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:24:43.272913 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:24:43.273018 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 05:24:43.273095 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0812 05:24:43.273192 (+ 97us) server_negotiation.cc:299] Negotiation successful
0812 05:24:43.273259 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260812 05:24:43.279243 4578 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 05:24:43.279733 4578 catalog_manager.cc:1504] Loading token signing keys...
I20260812 05:24:43.286178 4578 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 52df487fc568476081527b762b9ac395: Generated new TSK 0
I20260812 05:24:43.286332 4578 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 05:24:43.291899 32144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0waFvT/build/release/bin/kudu with pid 4459
2026-08-12T05:24:43Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1462 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786512017613366-32144-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 05:24:43 dist-test-slave-42z9 krb5kdc[4609](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 05:24:43 dist-test-slave-42z9 krb5kdc[4609](info): set up 2 sockets
Aug 12 05:24:43 dist-test-slave-42z9 krb5kdc[4609](info): commencing operation
krb5kdc: starting...
W20260812 05:24:44.451910 32144 mini_kdc.cc:121] Time spent starting KDC: real 1.129s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786512017613366-32144-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786512017613366-32144-0/krb5kdc/test-user.keytab.
Aug 12 05:24:44 dist-test-slave-42z9 krb5kdc[4609](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512284, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T05:24:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T05:24:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.100.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.100.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.100.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.62.keytab.
Entry for principal kudu/127.31.100.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.62.keytab.
I20260812 05:24:44.564029 32144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0waFvT/build/release/bin/kudu
/tmp/dist-test-task0waFvT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786512017613366-32144-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786512017613366-32144-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786512017613366-32144-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786512017613366-32144-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.100.62:45841
--webserver_interface=127.31.100.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.62.keytab
--principal=kudu/127.31.100.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.100.20:36311
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.100.62:45841
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786512017613366-32144-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786512017613366-32144-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786512017613366-32144-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 05:24:44.643164 4629 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 05:24:44.643353 4629 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 05:24:44.643383 4629 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 05:24:44.644469 4629 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 05:24:44.644510 4629 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 05:24:44.644527 4629 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 05:24:44.644544 4629 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 05:24:44.644560 4629 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 05:24:44.646059 4629 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.100.20:36311
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786512017613366-32144-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786512017613366-32144-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.100.62:45841
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.62.keytab
--principal=kudu/127.31.100.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.100.62:45841
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786512017613366-32144-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.100.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4629
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786512017613366-32144-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 04:20:34 UTC on ccycloud.quasar-phqdqp.root.comops.site
build id 4581871
I20260812 05:24:44.646281 4629 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 05:24:44.646481 4629 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 05:24:44.651160 4640 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 05:24:44.651177 4629 server_base.cc:1034] running on GCE node
W20260812 05:24:44.651152 4647 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 05:24:44.651152 4641 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 05:24:44.651680 4629 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 05:24:44.651922 4629 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 05:24:44.653053 4629 hybrid_clock.cc:648] HybridClock initialized: now 1786512284653039 us; error 30 us; skew 500 ppm
Aug 12 05:24:44 dist-test-slave-42z9 krb5kdc[4609](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512284, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.100.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 05:24:44.658473 4629 init.cc:377] Logged in from keytab as kudu/127.31.100.62@KRBTEST.COM (short username kudu)
I20260812 05:24:44.658854 4629 webserver.cc:460] Webserver started at http://127.31.100.62:46597/ using document root <none> and password file <none>
I20260812 05:24:44.659045 4629 fs_manager.cc:359] Metadata directory not provided
I20260812 05:24:44.659083 4629 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 05:24:44.659178 4629 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 05:24:44.660136 4629 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786512017613366-32144-0/minicluster-data/master-0/data/instance:
uuid: "8f84770d6328424393b49ff487e8750c"
format_stamp: "Formatted at 2026-08-12 05:24:44 on dist-test-slave-42z9"
server_key: "ec95812e1254f089fbc71b9d0547d155"
server_key_iv: "ee6369cbed7081f083b5045f1ce00234"
server_key_version: "encryptionkey@0"
I20260812 05:24:44.660418 4629 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786512017613366-32144-0/minicluster-data/master-0/wal/instance:
uuid: "8f84770d6328424393b49ff487e8750c"
format_stamp: "Formatted at 2026-08-12 05:24:44 on dist-test-slave-42z9"
server_key: "ec95812e1254f089fbc71b9d0547d155"
server_key_iv: "ee6369cbed7081f083b5045f1ce00234"
server_key_version: "encryptionkey@0"
I20260812 05:24:44.661548 4629 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260812 05:24:44.662195 4655 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 05:24:44.662328 4629 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260812 05:24:44.662412 4629 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786512017613366-32144-0/minicluster-data/master-0/data,/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786512017613366-32144-0/minicluster-data/master-0/wal
uuid: "8f84770d6328424393b49ff487e8750c"
format_stamp: "Formatted at 2026-08-12 05:24:44 on dist-test-slave-42z9"
server_key: "ec95812e1254f089fbc71b9d0547d155"
server_key_iv: "ee6369cbed7081f083b5045f1ce00234"
server_key_version: "encryptionkey@0"
I20260812 05:24:44.662469 4629 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786512017613366-32144-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786512017613366-32144-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786512017613366-32144-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 05:24:44.679903 4629 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 05:24:44.681497 4629 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 05:24:44.681631 4629 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 05:24:44.685250 4629 rpc_server.cc:307] RPC server started. Bound to: 127.31.100.62:45841
I20260812 05:24:44.685295 4743 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.100.62:45841 every 8 connection(s)
I20260812 05:24:44.685640 4629 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786512017613366-32144-0/minicluster-data/master-0/data/info.pb
I20260812 05:24:44.686445 4744 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 05:24:44.688011 32144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0waFvT/build/release/bin/kudu as pid 4629
I20260812 05:24:44.688125 32144 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786512017613366-32144-0/minicluster-data/master-0/wal/instance
I20260812 05:24:44.688282 32144 external_mini_cluster.cc:1428] Setting key c6bfab04387edaa3d1ed31b72f6dfb7f
I20260812 05:24:44.688699 4744 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8f84770d6328424393b49ff487e8750c: Bootstrap starting.
I20260812 05:24:44.689325 4744 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8f84770d6328424393b49ff487e8750c: Neither blocks nor log segments found. Creating new log.
I20260812 05:24:44.689682 4744 log.cc:824] T 00000000000000000000000000000000 P 8f84770d6328424393b49ff487e8750c: Log is configured to *not* fsync() on all Append() calls
I20260812 05:24:44.690367 4744 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8f84770d6328424393b49ff487e8750c: No bootstrap required, opened a new log
I20260812 05:24:44.692515 4744 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8f84770d6328424393b49ff487e8750c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f84770d6328424393b49ff487e8750c" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 45841 } }
I20260812 05:24:44.692677 4744 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8f84770d6328424393b49ff487e8750c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 05:24:44.692723 4744 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8f84770d6328424393b49ff487e8750c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f84770d6328424393b49ff487e8750c, State: Initialized, Role: FOLLOWER
I20260812 05:24:44.692883 4744 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8f84770d6328424393b49ff487e8750c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f84770d6328424393b49ff487e8750c" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 45841 } }
I20260812 05:24:44.692952 4744 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8f84770d6328424393b49ff487e8750c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 05:24:44.692981 4744 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8f84770d6328424393b49ff487e8750c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 05:24:44.693027 4744 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8f84770d6328424393b49ff487e8750c [term 0 FOLLOWER]: Advancing to term 1
I20260812 05:24:44.693461 4747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:24:44.688759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35943 (local address 127.31.100.62:45841)
0812 05:24:44.688938 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0812 05:24:44.688945 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 05:24:44.688969 (+ 24us) server_negotiation.cc:407] Connection header received
0812 05:24:44.689385 (+ 416us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:24:44.689391 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:24:44.689468 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:24:44.689589 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:24:44.689820 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:24:44.690713 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:24:44.691751 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:24:44.692119 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:24:44.692191 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:24:44.692803 (+ 612us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 05:24:44.692807 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 05:24:44.692808 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 05:24:44.692999 (+ 191us) server_negotiation.cc:922] calling sasl_server_start()
0812 05:24:44.693038 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 05:24:44.693053 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 05:24:44.693114 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0812 05:24:44.693184 (+ 70us) server_negotiation.cc:299] Negotiation successful
0812 05:24:44.693233 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":88,"threads_started":1}
I20260812 05:24:44.693747 4744 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8f84770d6328424393b49ff487e8750c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f84770d6328424393b49ff487e8750c" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 45841 } }
I20260812 05:24:44.693846 4744 leader_election.cc:302] T 00000000000000000000000000000000 P 8f84770d6328424393b49ff487e8750c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8f84770d6328424393b49ff487e8750c; no voters:
I20260812 05:24:44.693997 4744 leader_election.cc:288] T 00000000000000000000000000000000 P 8f84770d6328424393b49ff487e8750c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 05:24:44.694020 4752 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8f84770d6328424393b49ff487e8750c [term 1 FOLLOWER]: Leader election won for term 1
I20260812 05:24:44.694144 4752 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8f84770d6328424393b49ff487e8750c [term 1 LEADER]: Becoming Leader. State: Replica: 8f84770d6328424393b49ff487e8750c, State: Running, Role: LEADER
I20260812 05:24:44.694236 4744 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8f84770d6328424393b49ff487e8750c [sys.catalog]: configured and running, proceeding with master startup.
I20260812 05:24:44.694219 4752 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8f84770d6328424393b49ff487e8750c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f84770d6328424393b49ff487e8750c" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 45841 } }
I20260812 05:24:44.694341 32144 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 05:24:44.694542 4752 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f84770d6328424393b49ff487e8750c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8f84770d6328424393b49ff487e8750c. Latest consensus state: current_term: 1 leader_uuid: "8f84770d6328424393b49ff487e8750c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f84770d6328424393b49ff487e8750c" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 45841 } } }
I20260812 05:24:44.694618 4752 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f84770d6328424393b49ff487e8750c [sys.catalog]: This master's current role is: LEADER
I20260812 05:24:44.694823 4760 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 05:24:44.695482 4760 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 05:24:44.695866 4753 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f84770d6328424393b49ff487e8750c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8f84770d6328424393b49ff487e8750c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f84770d6328424393b49ff487e8750c" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 45841 } } }
I20260812 05:24:44.695957 4753 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f84770d6328424393b49ff487e8750c [sys.catalog]: This master's current role is: LEADER
I20260812 05:24:44.696925 4760 catalog_manager.cc:1347] Generated new cluster ID: acb126300ed948ed822e956573244afe
I20260812 05:24:44.696969 4760 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 05:24:44.701267 4747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:24:44.697816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47825 (local address 127.31.100.62:45841)
0812 05:24:44.697853 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0812 05:24:44.697854 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0812 05:24:44.697976 (+ 122us) server_negotiation.cc:407] Connection header received
0812 05:24:44.698087 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:24:44.698088 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:24:44.698098 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:24:44.698145 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:24:44.698284 (+ 139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:24:44.699100 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:24:44.700104 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:24:44.700394 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:24:44.700446 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:24:44.701060 (+ 614us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 05:24:44.701061 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 05:24:44.701061 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 05:24:44.701065 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0812 05:24:44.701080 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 05:24:44.701083 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 05:24:44.701123 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0812 05:24:44.701178 (+ 55us) server_negotiation.cc:299] Negotiation successful
0812 05:24:44.701199 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260812 05:24:44.708384 4760 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 05:24:44.708840 4760 catalog_manager.cc:1504] Loading token signing keys...
I20260812 05:24:44.711534 4760 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8f84770d6328424393b49ff487e8750c: Generated new TSK 0
I20260812 05:24:44.711692 4760 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 05:24:44.714231 32144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0waFvT/build/release/bin/kudu with pid 4629
2026-08-12T05:24:44Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1418 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786512017613366-32144-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 05:24:44 dist-test-slave-42z9 krb5kdc[4787](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 05:24:44 dist-test-slave-42z9 krb5kdc[4787](info): set up 2 sockets
Aug 12 05:24:44 dist-test-slave-42z9 krb5kdc[4787](info): commencing operation
krb5kdc: starting...
W20260812 05:24:45.871435 32144 mini_kdc.cc:121] Time spent starting KDC: real 1.130s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786512017613366-32144-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786512017613366-32144-0/krb5kdc/test-user.keytab.
Aug 12 05:24:45 dist-test-slave-42z9 krb5kdc[4787](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512285, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T05:24:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T05:24:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.100.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.100.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.100.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.62.keytab.
Entry for principal kudu/127.31.100.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.62.keytab.
I20260812 05:24:45.981851 32144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0waFvT/build/release/bin/kudu
/tmp/dist-test-task0waFvT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786512017613366-32144-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786512017613366-32144-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786512017613366-32144-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786512017613366-32144-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.100.62:42225
--webserver_interface=127.31.100.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.62.keytab
--principal=kudu/127.31.100.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.100.20:33929
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.100.62:42225
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786512017613366-32144-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786512017613366-32144-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786512017613366-32144-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 05:24:46.063561 4808 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 05:24:46.063776 4808 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 05:24:46.063799 4808 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 05:24:46.064958 4808 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 05:24:46.065001 4808 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 05:24:46.065013 4808 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 05:24:46.065025 4808 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 05:24:46.065050 4808 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 05:24:46.066834 4808 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.100.20:33929
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786512017613366-32144-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786512017613366-32144-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.100.62:42225
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.62.keytab
--principal=kudu/127.31.100.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.100.62:42225
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786512017613366-32144-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.100.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4808
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786512017613366-32144-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 04:20:34 UTC on ccycloud.quasar-phqdqp.root.comops.site
build id 4581871
I20260812 05:24:46.067102 4808 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 05:24:46.067318 4808 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 05:24:46.072033 4817 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 05:24:46.072024 4814 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 05:24:46.072053 4808 server_base.cc:1034] running on GCE node
W20260812 05:24:46.072023 4823 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 05:24:46.072461 4808 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 05:24:46.072767 4808 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 05:24:46.073905 4808 hybrid_clock.cc:648] HybridClock initialized: now 1786512286073890 us; error 30 us; skew 500 ppm
Aug 12 05:24:46 dist-test-slave-42z9 krb5kdc[4787](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512286, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.100.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 05:24:46.076344 4808 init.cc:377] Logged in from keytab as kudu/127.31.100.62@KRBTEST.COM (short username kudu)
I20260812 05:24:46.076901 4808 webserver.cc:460] Webserver started at http://127.31.100.62:32773/ using document root <none> and password file <none>
I20260812 05:24:46.077137 4808 fs_manager.cc:359] Metadata directory not provided
I20260812 05:24:46.077188 4808 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 05:24:46.077315 4808 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 05:24:46.078438 4808 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786512017613366-32144-0/minicluster-data/master-0/data/instance:
uuid: "5599f34b0d23477eaa31b352736cab0b"
format_stamp: "Formatted at 2026-08-12 05:24:46 on dist-test-slave-42z9"
server_key: "74dc9302245dba520b673d751d45c861"
server_key_iv: "2da0257b9c326530bbe1a92b36c84764"
server_key_version: "encryptionkey@0"
I20260812 05:24:46.078815 4808 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786512017613366-32144-0/minicluster-data/master-0/wal/instance:
uuid: "5599f34b0d23477eaa31b352736cab0b"
format_stamp: "Formatted at 2026-08-12 05:24:46 on dist-test-slave-42z9"
server_key: "74dc9302245dba520b673d751d45c861"
server_key_iv: "2da0257b9c326530bbe1a92b36c84764"
server_key_version: "encryptionkey@0"
I20260812 05:24:46.080061 4808 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260812 05:24:46.080726 4837 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 05:24:46.080876 4808 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260812 05:24:46.080962 4808 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786512017613366-32144-0/minicluster-data/master-0/data,/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786512017613366-32144-0/minicluster-data/master-0/wal
uuid: "5599f34b0d23477eaa31b352736cab0b"
format_stamp: "Formatted at 2026-08-12 05:24:46 on dist-test-slave-42z9"
server_key: "74dc9302245dba520b673d751d45c861"
server_key_iv: "2da0257b9c326530bbe1a92b36c84764"
server_key_version: "encryptionkey@0"
I20260812 05:24:46.081027 4808 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786512017613366-32144-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786512017613366-32144-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786512017613366-32144-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 05:24:46.084168 4808 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 05:24:46.086397 4808 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 05:24:46.086529 4808 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 05:24:46.090282 4808 rpc_server.cc:307] RPC server started. Bound to: 127.31.100.62:42225
I20260812 05:24:46.090324 4925 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.100.62:42225 every 8 connection(s)
I20260812 05:24:46.090617 4808 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786512017613366-32144-0/minicluster-data/master-0/data/info.pb
I20260812 05:24:46.091547 4926 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 05:24:46.093753 4926 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5599f34b0d23477eaa31b352736cab0b: Bootstrap starting.
I20260812 05:24:46.094295 4926 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5599f34b0d23477eaa31b352736cab0b: Neither blocks nor log segments found. Creating new log.
I20260812 05:24:46.094566 4926 log.cc:824] T 00000000000000000000000000000000 P 5599f34b0d23477eaa31b352736cab0b: Log is configured to *not* fsync() on all Append() calls
I20260812 05:24:46.095222 4926 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5599f34b0d23477eaa31b352736cab0b: No bootstrap required, opened a new log
I20260812 05:24:46.096136 32144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0waFvT/build/release/bin/kudu as pid 4808
I20260812 05:24:46.096249 32144 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786512017613366-32144-0/minicluster-data/master-0/wal/instance
I20260812 05:24:46.096441 32144 external_mini_cluster.cc:1428] Setting key 5ef6b9280e779078214d175f376fe24b
I20260812 05:24:46.097180 4926 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5599f34b0d23477eaa31b352736cab0b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5599f34b0d23477eaa31b352736cab0b" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 42225 } }
I20260812 05:24:46.097316 4926 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5599f34b0d23477eaa31b352736cab0b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 05:24:46.097347 4926 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5599f34b0d23477eaa31b352736cab0b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5599f34b0d23477eaa31b352736cab0b, State: Initialized, Role: FOLLOWER
I20260812 05:24:46.097544 4926 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5599f34b0d23477eaa31b352736cab0b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5599f34b0d23477eaa31b352736cab0b" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 42225 } }
I20260812 05:24:46.097659 4926 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5599f34b0d23477eaa31b352736cab0b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 05:24:46.097698 4926 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5599f34b0d23477eaa31b352736cab0b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 05:24:46.097748 4926 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5599f34b0d23477eaa31b352736cab0b [term 0 FOLLOWER]: Advancing to term 1
I20260812 05:24:46.098443 4932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:24:46.096929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55267 (local address 127.31.100.62:42225)
0812 05:24:46.097101 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0812 05:24:46.097110 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 05:24:46.097135 (+ 25us) server_negotiation.cc:407] Connection header received
0812 05:24:46.097224 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:24:46.097229 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:24:46.097251 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:24:46.097373 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:24:46.098036 (+ 663us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 05:24:46.098039 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 05:24:46.098040 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 05:24:46.098043 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0812 05:24:46.098085 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 05:24:46.098096 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 05:24:46.098151 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0812 05:24:46.098220 (+ 69us) server_negotiation.cc:299] Negotiation successful
0812 05:24:46.098259 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":79,"threads_started":1}
I20260812 05:24:46.098620 4926 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5599f34b0d23477eaa31b352736cab0b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5599f34b0d23477eaa31b352736cab0b" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 42225 } }
I20260812 05:24:46.098729 4926 leader_election.cc:302] T 00000000000000000000000000000000 P 5599f34b0d23477eaa31b352736cab0b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5599f34b0d23477eaa31b352736cab0b; no voters:
I20260812 05:24:46.098865 4926 leader_election.cc:288] T 00000000000000000000000000000000 P 5599f34b0d23477eaa31b352736cab0b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 05:24:46.098908 4934 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5599f34b0d23477eaa31b352736cab0b [term 1 FOLLOWER]: Leader election won for term 1
I20260812 05:24:46.099146 4926 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5599f34b0d23477eaa31b352736cab0b [sys.catalog]: configured and running, proceeding with master startup.
I20260812 05:24:46.099159 4934 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5599f34b0d23477eaa31b352736cab0b [term 1 LEADER]: Becoming Leader. State: Replica: 5599f34b0d23477eaa31b352736cab0b, State: Running, Role: LEADER
I20260812 05:24:46.099368 32144 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 05:24:46.100677 4934 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5599f34b0d23477eaa31b352736cab0b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5599f34b0d23477eaa31b352736cab0b" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 42225 } }
I20260812 05:24:46.101259 4938 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5599f34b0d23477eaa31b352736cab0b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5599f34b0d23477eaa31b352736cab0b. Latest consensus state: current_term: 1 leader_uuid: "5599f34b0d23477eaa31b352736cab0b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5599f34b0d23477eaa31b352736cab0b" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 42225 } } }
I20260812 05:24:46.101279 4936 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5599f34b0d23477eaa31b352736cab0b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5599f34b0d23477eaa31b352736cab0b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5599f34b0d23477eaa31b352736cab0b" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 42225 } } }
W20260812 05:24:46.101727 4971 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5599f34b0d23477eaa31b352736cab0b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 05:24:46.102017 4971 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 05:24:46.101991 4936 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5599f34b0d23477eaa31b352736cab0b [sys.catalog]: This master's current role is: LEADER
I20260812 05:24:46.102219 4974 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 05:24:46.101971 4938 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5599f34b0d23477eaa31b352736cab0b [sys.catalog]: This master's current role is: LEADER
I20260812 05:24:46.102412 4974 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 05:24:46.102638 4932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:24:46.100839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55375 (local address 127.31.100.62:42225)
0812 05:24:46.100870 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0812 05:24:46.100872 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 05:24:46.101065 (+ 193us) server_negotiation.cc:407] Connection header received
0812 05:24:46.101299 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:24:46.101299 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:24:46.101305 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:24:46.101335 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:24:46.102414 (+ 1079us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 05:24:46.102415 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 05:24:46.102415 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 05:24:46.102419 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0812 05:24:46.102433 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 05:24:46.102437 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 05:24:46.102467 (+ 30us) server_negotiation.cc:1035] Waiting for connection context
0812 05:24:46.102572 (+ 105us) server_negotiation.cc:299] Negotiation successful
0812 05:24:46.102585 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260812 05:24:46.103854 4974 catalog_manager.cc:1347] Generated new cluster ID: 50bde63ef9b547c3a6b92411a6683461
I20260812 05:24:46.103905 4974 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 05:24:46.115279 4974 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 05:24:46.115414 4974 catalog_manager.cc:1504] Loading token signing keys...
I20260812 05:24:46.121342 4974 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5599f34b0d23477eaa31b352736cab0b: Generated new TSK 0
I20260812 05:24:46.121532 4974 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 05:24:46.126652 32144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0waFvT/build/release/bin/kudu with pid 4808
2026-08-12T05:24:46Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1416 ms)
[----------] 18 tests from AuthTokenIssuingTest (25884 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786512017613366-32144-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 05:24:46 dist-test-slave-42z9 krb5kdc[4978](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 05:24:46 dist-test-slave-42z9 krb5kdc[4978](info): set up 2 sockets
Aug 12 05:24:46 dist-test-slave-42z9 krb5kdc[4978](info): commencing operation
krb5kdc: starting...
W20260812 05:24:47.311286 32144 mini_kdc.cc:121] Time spent starting KDC: real 1.153s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786512017613366-32144-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786512017613366-32144-0/krb5kdc/test-user.keytab.
Aug 12 05:24:47 dist-test-slave-42z9 krb5kdc[4978](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512287, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T05:24:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T05:24:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.100.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.100.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.100.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.62.keytab.
Entry for principal kudu/127.31.100.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.62.keytab.
I20260812 05:24:47.423321 32144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0waFvT/build/release/bin/kudu
/tmp/dist-test-task0waFvT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786512017613366-32144-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786512017613366-32144-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786512017613366-32144-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786512017613366-32144-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.100.62:37187
--webserver_interface=127.31.100.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.62.keytab
--principal=kudu/127.31.100.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.100.20:34669
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.100.62:37187,127.31.100.61:44651,127.31.100.60:43459
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786512017613366-32144-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786512017613366-32144-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786512017613366-32144-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 05:24:47.501853 4997 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260812 05:24:47.502054 4997 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 05:24:47.502071 4997 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 05:24:47.502089 4997 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 05:24:47.503154 4997 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 05:24:47.503188 4997 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 05:24:47.503201 4997 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 05:24:47.503211 4997 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 05:24:47.503224 4997 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 05:24:47.504688 4997 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.100.20:34669
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786512017613366-32144-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786512017613366-32144-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.100.62:37187,127.31.100.61:44651,127.31.100.60:43459
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.62.keytab
--principal=kudu/127.31.100.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.100.62:37187
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786512017613366-32144-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.100.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4997
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786512017613366-32144-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 04:20:34 UTC on ccycloud.quasar-phqdqp.root.comops.site
build id 4581871
I20260812 05:24:47.504876 4997 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 05:24:47.505040 4997 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 05:24:47.509598 5011 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 05:24:47.509644 5010 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 05:24:47.509723 4997 server_base.cc:1034] running on GCE node
W20260812 05:24:47.509608 5017 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 05:24:47.510092 4997 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 05:24:47.510342 4997 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 05:24:47.511504 4997 hybrid_clock.cc:648] HybridClock initialized: now 1786512287511486 us; error 33 us; skew 500 ppm
Aug 12 05:24:47 dist-test-slave-42z9 krb5kdc[4978](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512287, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.100.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 05:24:47.514016 4997 init.cc:377] Logged in from keytab as kudu/127.31.100.62@KRBTEST.COM (short username kudu)
I20260812 05:24:47.514518 4997 webserver.cc:460] Webserver started at http://127.31.100.62:45639/ using document root <none> and password file <none>
I20260812 05:24:47.514757 4997 fs_manager.cc:359] Metadata directory not provided
I20260812 05:24:47.514822 4997 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 05:24:47.514963 4997 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 05:24:47.516336 4997 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786512017613366-32144-0/minicluster-data/master-0/data/instance:
uuid: "4e24c865796649b6bffc44f5c61d57f3"
format_stamp: "Formatted at 2026-08-12 05:24:47 on dist-test-slave-42z9"
server_key: "609fd90914a52a3b3a0a89439b6a2d59"
server_key_iv: "55910b3ea059006fd8dc7ab38143b7b9"
server_key_version: "encryptionkey@0"
I20260812 05:24:47.516855 4997 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786512017613366-32144-0/minicluster-data/master-0/wal/instance:
uuid: "4e24c865796649b6bffc44f5c61d57f3"
format_stamp: "Formatted at 2026-08-12 05:24:47 on dist-test-slave-42z9"
server_key: "609fd90914a52a3b3a0a89439b6a2d59"
server_key_iv: "55910b3ea059006fd8dc7ab38143b7b9"
server_key_version: "encryptionkey@0"
I20260812 05:24:47.518178 4997 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260812 05:24:47.518823 5030 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 05:24:47.518944 4997 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260812 05:24:47.519033 4997 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786512017613366-32144-0/minicluster-data/master-0/data,/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786512017613366-32144-0/minicluster-data/master-0/wal
uuid: "4e24c865796649b6bffc44f5c61d57f3"
format_stamp: "Formatted at 2026-08-12 05:24:47 on dist-test-slave-42z9"
server_key: "609fd90914a52a3b3a0a89439b6a2d59"
server_key_iv: "55910b3ea059006fd8dc7ab38143b7b9"
server_key_version: "encryptionkey@0"
I20260812 05:24:47.519099 4997 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786512017613366-32144-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786512017613366-32144-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786512017613366-32144-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 05:24:47.567349 4997 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 05:24:47.569563 4997 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 05:24:47.569695 4997 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 05:24:47.573427 4997 rpc_server.cc:307] RPC server started. Bound to: 127.31.100.62:37187
I20260812 05:24:47.573503 5107 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.100.62:37187 every 8 connection(s)
I20260812 05:24:47.573828 4997 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786512017613366-32144-0/minicluster-data/master-0/data/info.pb
I20260812 05:24:47.574771 5108 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 05:24:47.576289 5108 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.100.62" port: 37187 } has no permanent_uuid. Determining permanent_uuid...
I20260812 05:24:47.577606 32144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0waFvT/build/release/bin/kudu as pid 4997
I20260812 05:24:47.577701 32144 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786512017613366-32144-0/minicluster-data/master-0/wal/instance
I20260812 05:24:47.577883 32144 external_mini_cluster.cc:1428] Setting key 4ab5f3233e8f00111020a369b1400773
Aug 12 05:24:47 dist-test-slave-42z9 krb5kdc[4978](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512287, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.100.62@KRBTEST.COM for kudu/127.31.100.62@KRBTEST.COM
I20260812 05:24:47.586877 5114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:24:47.576932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.62:37187 (local address 127.0.0.1:44592)
0812 05:24:47.577152 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0812 05:24:47.577191 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0812 05:24:47.577719 (+ 528us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:24:47.578047 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:24:47.578049 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:24:47.578422 (+ 373us) client_negotiation.cc:189] Negotiated authn=SASL
0812 05:24:47.578645 (+ 223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:24:47.578651 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:24:47.580019 (+ 1368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:24:47.580020 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:24:47.581066 (+ 1046us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:24:47.581069 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:24:47.581701 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:24:47.581703 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:24:47.581762 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:24:47.582188 (+ 426us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 05:24:47.582196 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0812 05:24:47.583709 (+ 1513us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 05:24:47.585555 (+ 1846us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 05:24:47.585561 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 05:24:47.585562 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 05:24:47.585743 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 05:24:47.585996 (+ 253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 05:24:47.585996 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 05:24:47.585996 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 05:24:47.586074 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 05:24:47.586480 (+ 406us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 05:24:47.586484 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 05:24:47.586537 (+ 53us) client_negotiation.cc:769] Sending connection context
0812 05:24:47.586606 (+ 69us) client_negotiation.cc:240] Negotiation successful
0812 05:24:47.586723 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":94,"threads_started":1}
I20260812 05:24:47.586900 5115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:24:47.577113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44592 (local address 127.31.100.62:37187)
0812 05:24:47.577232 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0812 05:24:47.577236 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 05:24:47.577266 (+ 30us) server_negotiation.cc:407] Connection header received
0812 05:24:47.577880 (+ 614us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:24:47.577883 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:24:47.577905 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:24:47.578012 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:24:47.578757 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:24:47.579910 (+ 1153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:24:47.581149 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:24:47.581591 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:24:47.581696 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:24:47.583797 (+ 2101us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 05:24:47.583807 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 05:24:47.583807 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 05:24:47.583815 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0812 05:24:47.585481 (+ 1666us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:24:47.585819 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:24:47.585821 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:24:47.585822 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:24:47.585922 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:24:47.586134 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:24:47.586135 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:24:47.586135 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:24:47.586411 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 05:24:47.586504 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0812 05:24:47.586660 (+ 156us) server_negotiation.cc:299] Negotiation successful
0812 05:24:47.586801 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":40,"server-negotiator.queue_time_us":89,"thread_start_us":44,"threads_started":1}
WARNING: no policy specified for kudu/127.31.100.61@KRBTEST.COM; defaulting to no policy
I20260812 05:24:47.587561 5108 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.100.61" port: 44651 } has no permanent_uuid. Determining permanent_uuid...
I20260812 05:24:47.587873 5114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:24:47.587767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.61:44651 (local address 127.0.0.1:38608)
0812 05:24:47.587812 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0812 05:24:47.587854 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.100.61:44651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260812 05:24:47.588022 5031 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.100.61:44651: connect: Connection refused (error 111)
I20260812 05:24:47.588377 5114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:24:47.588308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.61:44651 (local address 127.0.0.1:38610)
0812 05:24:47.588338 (+ 30us) negotiation.cc:107] Waiting for socket to connect
0812 05:24:47.588362 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.100.61:44651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":20}
W20260812 05:24:47.588497 5108 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.100.61:44651: Network error: Client connection negotiation failed: client connection to 127.31.100.61:44651: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.100.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.100.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.61.keytab.
Entry for principal kudu/127.31.100.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.61.keytab.
I20260812 05:24:47.602193 32144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0waFvT/build/release/bin/kudu
/tmp/dist-test-task0waFvT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786512017613366-32144-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786512017613366-32144-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786512017613366-32144-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786512017613366-32144-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.100.61:44651
--webserver_interface=127.31.100.61
--webserver_port=0
--keytab_file=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.61.keytab
--principal=kudu/127.31.100.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.100.20:34669
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.100.62:37187,127.31.100.61:44651,127.31.100.60:43459
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786512017613366-32144-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786512017613366-32144-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786512017613366-32144-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 05:24:47.637708 5108 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.100.61" port: 44651 } attempt: 1
I20260812 05:24:47.638173 5114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:24:47.638051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.61:44651 (local address 127.0.0.1:38622)
0812 05:24:47.638109 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0812 05:24:47.638142 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.100.61:44651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260812 05:24:47.638514 5114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:24:47.638429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.61:44651 (local address 127.0.0.1:38634)
0812 05:24:47.638470 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0812 05:24:47.638495 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.100.61:44651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260812 05:24:47.638604 5108 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.100.61:44651: Network error: Client connection negotiation failed: client connection to 127.31.100.61:44651: connect: Connection refused (error 111)
W20260812 05:24:47.682209 5122 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260812 05:24:47.682410 5122 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 05:24:47.682433 5122 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 05:24:47.682456 5122 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 05:24:47.683548 5122 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 05:24:47.683586 5122 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 05:24:47.683605 5122 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 05:24:47.683621 5122 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 05:24:47.683636 5122 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 05:24:47.685101 5122 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.100.20:34669
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786512017613366-32144-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786512017613366-32144-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.31.100.62:37187,127.31.100.61:44651,127.31.100.60:43459
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.61.keytab
--principal=kudu/127.31.100.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.100.61:44651
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786512017613366-32144-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.31.100.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5122
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786512017613366-32144-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 04:20:34 UTC on ccycloud.quasar-phqdqp.root.comops.site
build id 4581871
I20260812 05:24:47.685346 5122 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 05:24:47.685570 5122 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 05:24:47.690094 5137 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 05:24:47.690114 5134 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 05:24:47.690114 5139 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 05:24:47.690234 5122 server_base.cc:1034] running on GCE node
I20260812 05:24:47.690521 5122 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 05:24:47.690805 5122 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 05:24:47.691944 5122 hybrid_clock.cc:648] HybridClock initialized: now 1786512287691929 us; error 33 us; skew 500 ppm
Aug 12 05:24:47 dist-test-slave-42z9 krb5kdc[4978](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512287, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.100.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 05:24:47.694294 5122 init.cc:377] Logged in from keytab as kudu/127.31.100.61@KRBTEST.COM (short username kudu)
I20260812 05:24:47.694680 5122 webserver.cc:460] Webserver started at http://127.31.100.61:34363/ using document root <none> and password file <none>
I20260812 05:24:47.694887 5122 fs_manager.cc:359] Metadata directory not provided
I20260812 05:24:47.694934 5122 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 05:24:47.695071 5122 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 05:24:47.696056 5122 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786512017613366-32144-0/minicluster-data/master-1/data/instance:
uuid: "5fc02b8d8ab74e6896097081da53ef7f"
format_stamp: "Formatted at 2026-08-12 05:24:47 on dist-test-slave-42z9"
server_key: "e271a3b1b749ed311b49df2f531a6b2d"
server_key_iv: "4b2a9d2978955b6474af9b940d023372"
server_key_version: "encryptionkey@0"
I20260812 05:24:47.696377 5122 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786512017613366-32144-0/minicluster-data/master-1/wal/instance:
uuid: "5fc02b8d8ab74e6896097081da53ef7f"
format_stamp: "Formatted at 2026-08-12 05:24:47 on dist-test-slave-42z9"
server_key: "e271a3b1b749ed311b49df2f531a6b2d"
server_key_iv: "4b2a9d2978955b6474af9b940d023372"
server_key_version: "encryptionkey@0"
I20260812 05:24:47.697639 5122 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260812 05:24:47.698335 5150 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 05:24:47.698484 5122 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260812 05:24:47.698570 5122 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786512017613366-32144-0/minicluster-data/master-1/data,/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786512017613366-32144-0/minicluster-data/master-1/wal
uuid: "5fc02b8d8ab74e6896097081da53ef7f"
format_stamp: "Formatted at 2026-08-12 05:24:47 on dist-test-slave-42z9"
server_key: "e271a3b1b749ed311b49df2f531a6b2d"
server_key_iv: "4b2a9d2978955b6474af9b940d023372"
server_key_version: "encryptionkey@0"
I20260812 05:24:47.698633 5122 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786512017613366-32144-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786512017613366-32144-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786512017613366-32144-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 05:24:47.706727 5108 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.100.61" port: 44651 } attempt: 2
I20260812 05:24:47.707047 5114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:24:47.706945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.61:44651 (local address 127.0.0.1:38642)
0812 05:24:47.706995 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0812 05:24:47.707020 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.100.61:44651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260812 05:24:47.707316 5114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:24:47.707244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.61:44651 (local address 127.0.0.1:38656)
0812 05:24:47.707277 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0812 05:24:47.707299 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.100.61:44651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
W20260812 05:24:47.707391 5108 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.100.61:44651: Network error: Client connection negotiation failed: client connection to 127.31.100.61:44651: connect: Connection refused (error 111)
I20260812 05:24:47.716269 5122 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 05:24:47.717880 5122 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 05:24:47.718006 5122 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 05:24:47.721695 5122 rpc_server.cc:307] RPC server started. Bound to: 127.31.100.61:44651
I20260812 05:24:47.721732 5236 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.100.61:44651 every 8 connection(s)
I20260812 05:24:47.722031 5122 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786512017613366-32144-0/minicluster-data/master-1/data/info.pb
I20260812 05:24:47.722942 5237 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 05:24:47.724372 5237 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.100.62" port: 37187 } has no permanent_uuid. Determining permanent_uuid...
I20260812 05:24:47.726333 32144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0waFvT/build/release/bin/kudu as pid 5122
I20260812 05:24:47.726433 32144 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786512017613366-32144-0/minicluster-data/master-1/wal/instance
I20260812 05:24:47.726605 32144 external_mini_cluster.cc:1428] Setting key c85b899b9d63c71b3163f50579304107
Aug 12 05:24:47 dist-test-slave-42z9 krb5kdc[4978](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512287, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.100.61@KRBTEST.COM for kudu/127.31.100.62@KRBTEST.COM
I20260812 05:24:47.732870 5115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:24:47.724989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44606 (local address 127.31.100.62:37187)
0812 05:24:47.725041 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0812 05:24:47.725043 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 05:24:47.725228 (+ 185us) server_negotiation.cc:407] Connection header received
0812 05:24:47.725738 (+ 510us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:24:47.725739 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:24:47.725750 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:24:47.725790 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:24:47.726342 (+ 552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:24:47.727027 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:24:47.727931 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:24:47.728233 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:24:47.728303 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:24:47.730390 (+ 2087us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 05:24:47.730392 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 05:24:47.730392 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 05:24:47.730399 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 05:24:47.731648 (+ 1249us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:24:47.732012 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:24:47.732013 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:24:47.732013 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:24:47.732069 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:24:47.732286 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:24:47.732287 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:24:47.732287 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:24:47.732395 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 05:24:47.732449 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0812 05:24:47.732701 (+ 252us) server_negotiation.cc:299] Negotiation successful
0812 05:24:47.732788 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260812 05:24:47.733008 5241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:24:47.724952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.62:37187 (local address 127.0.0.1:44606)
0812 05:24:47.725120 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0812 05:24:47.725157 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0812 05:24:47.725642 (+ 485us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:24:47.725857 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:24:47.725860 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:24:47.726091 (+ 231us) client_negotiation.cc:189] Negotiated authn=SASL
0812 05:24:47.726266 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:24:47.726270 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:24:47.727114 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:24:47.727114 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:24:47.727851 (+ 737us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:24:47.727853 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:24:47.728333 (+ 480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:24:47.728334 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:24:47.728465 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:24:47.728876 (+ 411us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 05:24:47.728885 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0812 05:24:47.730302 (+ 1417us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 05:24:47.731724 (+ 1422us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 05:24:47.731731 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 05:24:47.731733 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 05:24:47.731945 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 05:24:47.732128 (+ 183us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 05:24:47.732129 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 05:24:47.732129 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 05:24:47.732230 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 05:24:47.732449 (+ 219us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 05:24:47.732456 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 05:24:47.732583 (+ 127us) client_negotiation.cc:769] Sending connection context
0812 05:24:47.732667 (+ 84us) client_negotiation.cc:240] Negotiation successful
0812 05:24:47.732807 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":85,"threads_started":1}
I20260812 05:24:47.733690 5237 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.100.61" port: 44651 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.31.100.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.100.60@KRBTEST.COM" created.
Aug 12 05:24:47 dist-test-slave-42z9 krb5kdc[4978](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512287, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.100.61@KRBTEST.COM for kudu/127.31.100.61@KRBTEST.COM
I20260812 05:24:47.740320 5241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:24:47.733943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.61:44651 (local address 127.0.0.1:38658)
0812 05:24:47.733984 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0812 05:24:47.733993 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0812 05:24:47.734027 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:24:47.734253 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:24:47.734255 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:24:47.734383 (+ 128us) client_negotiation.cc:189] Negotiated authn=SASL
0812 05:24:47.734434 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:24:47.734435 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:24:47.735251 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:24:47.735252 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:24:47.735758 (+ 506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:24:47.735760 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:24:47.736145 (+ 385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:24:47.736146 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:24:47.736192 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:24:47.736479 (+ 287us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 05:24:47.736482 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0812 05:24:47.737801 (+ 1319us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 05:24:47.739336 (+ 1535us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 05:24:47.739337 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 05:24:47.739337 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 05:24:47.739562 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 05:24:47.739737 (+ 175us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 05:24:47.739737 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 05:24:47.739738 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 05:24:47.739809 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 05:24:47.740029 (+ 220us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 05:24:47.740030 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 05:24:47.740074 (+ 44us) client_negotiation.cc:769] Sending connection context
0812 05:24:47.740146 (+ 72us) client_negotiation.cc:240] Negotiation successful
0812 05:24:47.740238 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260812 05:24:47.740463 5243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:24:47.733981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38658 (local address 127.31.100.61:44651)
0812 05:24:47.734117 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0812 05:24:47.734121 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 05:24:47.734139 (+ 18us) server_negotiation.cc:407] Connection header received
0812 05:24:47.734157 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:24:47.734159 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:24:47.734169 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:24:47.734229 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:24:47.734506 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:24:47.735178 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:24:47.735813 (+ 635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:24:47.736091 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:24:47.736143 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:24:47.737880 (+ 1737us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 05:24:47.737889 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 05:24:47.737889 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 05:24:47.737897 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0812 05:24:47.739259 (+ 1362us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:24:47.739633 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:24:47.739635 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:24:47.739635 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:24:47.739682 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:24:47.739862 (+ 180us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:24:47.739863 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:24:47.739863 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:24:47.739971 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 05:24:47.740024 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0812 05:24:47.740175 (+ 151us) server_negotiation.cc:299] Negotiation successful
0812 05:24:47.740325 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":69,"server-negotiator.queue_time_us":110,"thread_start_us":74,"threads_started":1}
I20260812 05:24:47.740985 5237 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.100.60" port: 43459 } has no permanent_uuid. Determining permanent_uuid...
I20260812 05:24:47.741287 5241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:24:47.741173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.60:43459 (local address 127.0.0.1:43736)
0812 05:24:47.741214 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0812 05:24:47.741264 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.100.60:43459: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260812 05:24:47.741427 5157 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.100.60:43459: connect: Connection refused (error 111)
I20260812 05:24:47.741844 5241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:24:47.741749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.60:43459 (local address 127.0.0.1:43746)
0812 05:24:47.741793 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0812 05:24:47.741823 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.100.60:43459: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260812 05:24:47.741945 5237 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.100.60:43459: Network error: Client connection negotiation failed: client connection to 127.31.100.60:43459: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.100.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.60.keytab.
Entry for principal kudu/127.31.100.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.60.keytab.
I20260812 05:24:47.750252 32144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0waFvT/build/release/bin/kudu
/tmp/dist-test-task0waFvT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786512017613366-32144-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786512017613366-32144-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786512017613366-32144-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786512017613366-32144-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.100.60:43459
--webserver_interface=127.31.100.60
--webserver_port=0
--keytab_file=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.60.keytab
--principal=kudu/127.31.100.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.100.20:34669
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.100.62:37187,127.31.100.61:44651,127.31.100.60:43459
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786512017613366-32144-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786512017613366-32144-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786512017613366-32144-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 05:24:47.791158 5237 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.100.60" port: 43459 } attempt: 1
I20260812 05:24:47.791594 5241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:24:47.791462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.60:43459 (local address 127.0.0.1:43754)
0812 05:24:47.791520 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0812 05:24:47.791562 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.100.60:43459: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260812 05:24:47.791887 5241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:24:47.791805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.60:43459 (local address 127.0.0.1:43756)
0812 05:24:47.791844 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0812 05:24:47.791868 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.100.60:43459: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260812 05:24:47.791968 5237 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.100.60:43459: Network error: Client connection negotiation failed: client connection to 127.31.100.60:43459: connect: Connection refused (error 111)
I20260812 05:24:47.798516 5108 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.100.61" port: 44651 } attempt: 3
Aug 12 05:24:47 dist-test-slave-42z9 krb5kdc[4978](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512287, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.100.62@KRBTEST.COM for kudu/127.31.100.61@KRBTEST.COM
I20260812 05:24:47.805716 5243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:24:47.798826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38660 (local address 127.31.100.61:44651)
0812 05:24:47.798880 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0812 05:24:47.798881 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0812 05:24:47.798891 (+ 10us) server_negotiation.cc:407] Connection header received
0812 05:24:47.798933 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:24:47.798934 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:24:47.798945 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:24:47.798991 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:24:47.799331 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:24:47.800109 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:24:47.800770 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:24:47.801131 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:24:47.801200 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:24:47.803292 (+ 2092us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 05:24:47.803293 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 05:24:47.803294 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 05:24:47.803302 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0812 05:24:47.804700 (+ 1398us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:24:47.805059 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:24:47.805060 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:24:47.805060 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:24:47.805107 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:24:47.805277 (+ 170us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:24:47.805278 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:24:47.805278 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:24:47.805367 (+ 89us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 05:24:47.805411 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0812 05:24:47.805586 (+ 175us) server_negotiation.cc:299] Negotiation successful
0812 05:24:47.805658 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260812 05:24:47.805758 5114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:24:47.798768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.61:44651 (local address 127.0.0.1:38660)
0812 05:24:47.798822 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0812 05:24:47.798832 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0812 05:24:47.798878 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:24:47.799001 (+ 123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:24:47.799002 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:24:47.799157 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0812 05:24:47.799238 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:24:47.799241 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:24:47.800199 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:24:47.800200 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:24:47.800698 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:24:47.800699 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:24:47.801201 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:24:47.801202 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:24:47.801266 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:24:47.801720 (+ 454us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 05:24:47.801724 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0812 05:24:47.803178 (+ 1454us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 05:24:47.804775 (+ 1597us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 05:24:47.804776 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 05:24:47.804776 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 05:24:47.804995 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 05:24:47.805163 (+ 168us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 05:24:47.805164 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 05:24:47.805164 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 05:24:47.805228 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 05:24:47.805420 (+ 192us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 05:24:47.805421 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 05:24:47.805494 (+ 73us) client_negotiation.cc:769] Sending connection context
0812 05:24:47.805568 (+ 74us) client_negotiation.cc:240] Negotiation successful
0812 05:24:47.805660 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260812 05:24:47.806121 5108 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.100.60" port: 43459 } has no permanent_uuid. Determining permanent_uuid...
I20260812 05:24:47.806458 5114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:24:47.806350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.60:43459 (local address 127.0.0.1:43762)
0812 05:24:47.806403 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0812 05:24:47.806435 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.100.60:43459: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260812 05:24:47.806790 5114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:24:47.806708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.60:43459 (local address 127.0.0.1:43764)
0812 05:24:47.806749 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0812 05:24:47.806773 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.100.60:43459: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260812 05:24:47.806877 5108 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.100.60:43459: Network error: Client connection negotiation failed: client connection to 127.31.100.60:43459: connect: Connection refused (error 111)
W20260812 05:24:47.831662 5248 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260812 05:24:47.831848 5248 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 05:24:47.831864 5248 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 05:24:47.831904 5248 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 05:24:47.833003 5248 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 05:24:47.833039 5248 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 05:24:47.833052 5248 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 05:24:47.833076 5248 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 05:24:47.833097 5248 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 05:24:47.834611 5248 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.100.20:34669
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786512017613366-32144-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786512017613366-32144-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.31.100.62:37187,127.31.100.61:44651,127.31.100.60:43459
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.60.keytab
--principal=kudu/127.31.100.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.100.60:43459
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786512017613366-32144-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.31.100.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5248
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786512017613366-32144-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 04:20:34 UTC on ccycloud.quasar-phqdqp.root.comops.site
build id 4581871
I20260812 05:24:47.834833 5248 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 05:24:47.835047 5248 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260812 05:24:47.838028 5108 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.100.60" port: 43459 } attempt: 1
I20260812 05:24:47.838506 5114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:24:47.838342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.60:43459 (local address 127.0.0.1:43774)
0812 05:24:47.838410 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0812 05:24:47.838477 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.100.60:43459: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260812 05:24:47.839018 5114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:24:47.838754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.60:43459 (local address 127.0.0.1:43778)
0812 05:24:47.838971 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0812 05:24:47.838998 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.100.60:43459: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200}
W20260812 05:24:47.839138 5108 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.100.60:43459: Network error: Client connection negotiation failed: client connection to 127.31.100.60:43459: connect: Connection refused (error 111)
W20260812 05:24:47.839823 5256 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 05:24:47.839841 5261 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 05:24:47.839834 5255 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 05:24:47.840160 5248 server_base.cc:1034] running on GCE node
I20260812 05:24:47.840375 5248 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 05:24:47.840643 5248 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 05:24:47.841776 5248 hybrid_clock.cc:648] HybridClock initialized: now 1786512287841763 us; error 26 us; skew 500 ppm
Aug 12 05:24:47 dist-test-slave-42z9 krb5kdc[4978](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512287, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.100.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 05:24:47.844113 5248 init.cc:377] Logged in from keytab as kudu/127.31.100.60@KRBTEST.COM (short username kudu)
I20260812 05:24:47.844515 5248 webserver.cc:460] Webserver started at http://127.31.100.60:44187/ using document root <none> and password file <none>
I20260812 05:24:47.844730 5248 fs_manager.cc:359] Metadata directory not provided
I20260812 05:24:47.844774 5248 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 05:24:47.844897 5248 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 05:24:47.845935 5248 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786512017613366-32144-0/minicluster-data/master-2/data/instance:
uuid: "66bc144962dc47d5aae05be95181bb55"
format_stamp: "Formatted at 2026-08-12 05:24:47 on dist-test-slave-42z9"
server_key: "c213a9f24c902912ef6f80691a5c17c7"
server_key_iv: "e893fe864471f08319b327d3c03baccd"
server_key_version: "encryptionkey@0"
I20260812 05:24:47.846249 5248 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786512017613366-32144-0/minicluster-data/master-2/wal/instance:
uuid: "66bc144962dc47d5aae05be95181bb55"
format_stamp: "Formatted at 2026-08-12 05:24:47 on dist-test-slave-42z9"
server_key: "c213a9f24c902912ef6f80691a5c17c7"
server_key_iv: "e893fe864471f08319b327d3c03baccd"
server_key_version: "encryptionkey@0"
I20260812 05:24:47.847497 5248 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260812 05:24:47.848152 5272 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 05:24:47.848299 5248 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260812 05:24:47.848382 5248 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786512017613366-32144-0/minicluster-data/master-2/data,/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786512017613366-32144-0/minicluster-data/master-2/wal
uuid: "66bc144962dc47d5aae05be95181bb55"
format_stamp: "Formatted at 2026-08-12 05:24:47 on dist-test-slave-42z9"
server_key: "c213a9f24c902912ef6f80691a5c17c7"
server_key_iv: "e893fe864471f08319b327d3c03baccd"
server_key_version: "encryptionkey@0"
I20260812 05:24:47.848466 5248 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786512017613366-32144-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786512017613366-32144-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786512017613366-32144-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 05:24:47.860093 5237 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.100.60" port: 43459 } attempt: 2
I20260812 05:24:47.860450 5241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:24:47.860330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.60:43459 (local address 127.0.0.1:43786)
0812 05:24:47.860383 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0812 05:24:47.860419 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.100.60:43459: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260812 05:24:47.860742 5241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:24:47.860675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.60:43459 (local address 127.0.0.1:43790)
0812 05:24:47.860704 (+ 29us) negotiation.cc:107] Waiting for socket to connect
0812 05:24:47.860726 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.100.60:43459: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
W20260812 05:24:47.860811 5237 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.100.60:43459: Network error: Client connection negotiation failed: client connection to 127.31.100.60:43459: connect: Connection refused (error 111)
I20260812 05:24:47.865375 5248 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 05:24:47.867014 5248 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 05:24:47.867120 5248 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 05:24:47.870762 5248 rpc_server.cc:307] RPC server started. Bound to: 127.31.100.60:43459
I20260812 05:24:47.870793 5355 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.100.60:43459 every 8 connection(s)
I20260812 05:24:47.871093 5248 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786512017613366-32144-0/minicluster-data/master-2/data/info.pb
I20260812 05:24:47.872009 5357 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 05:24:47.873502 5357 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.100.62" port: 37187 } has no permanent_uuid. Determining permanent_uuid...
I20260812 05:24:47.874429 32144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0waFvT/build/release/bin/kudu as pid 5248
I20260812 05:24:47.874533 32144 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786512017613366-32144-0/minicluster-data/master-2/wal/instance
I20260812 05:24:47.874718 32144 external_mini_cluster.cc:1428] Setting key e83983d866ba0338c545aa4330763ded
Aug 12 05:24:47 dist-test-slave-42z9 krb5kdc[4978](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512287, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.100.60@KRBTEST.COM for kudu/127.31.100.62@KRBTEST.COM
Aug 12 05:24:47 dist-test-slave-42z9 krb5kdc[4978](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512287, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.100.60@KRBTEST.COM
I20260812 05:24:47.882385 5115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:24:47.874147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44614 (local address 127.31.100.62:37187)
0812 05:24:47.874210 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0812 05:24:47.874212 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 05:24:47.874472 (+ 260us) server_negotiation.cc:407] Connection header received
0812 05:24:47.875069 (+ 597us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:24:47.875071 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:24:47.875083 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:24:47.875136 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:24:47.875790 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:24:47.876262 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:24:47.877249 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:24:47.877465 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:24:47.877528 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:24:47.879650 (+ 2122us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 05:24:47.879651 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 05:24:47.879651 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 05:24:47.879658 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 05:24:47.881148 (+ 1490us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:24:47.881532 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:24:47.881533 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:24:47.881534 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:24:47.881596 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:24:47.881813 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:24:47.881813 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:24:47.881814 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:24:47.881929 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 05:24:47.881976 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0812 05:24:47.882212 (+ 236us) server_negotiation.cc:299] Negotiation successful
0812 05:24:47.882294 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260812 05:24:47.882535 5359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:24:47.874118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.62:37187 (local address 127.0.0.1:44614)
0812 05:24:47.874304 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0812 05:24:47.874351 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0812 05:24:47.874966 (+ 615us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:24:47.875211 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:24:47.875215 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:24:47.875550 (+ 335us) client_negotiation.cc:189] Negotiated authn=SASL
0812 05:24:47.875712 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:24:47.875716 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:24:47.876329 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:24:47.876330 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:24:47.877173 (+ 843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:24:47.877175 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:24:47.877555 (+ 380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:24:47.877556 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:24:47.877676 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:24:47.878117 (+ 441us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 05:24:47.878127 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0812 05:24:47.879555 (+ 1428us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 05:24:47.881221 (+ 1666us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 05:24:47.881227 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 05:24:47.881229 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 05:24:47.881477 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 05:24:47.881666 (+ 189us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 05:24:47.881667 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 05:24:47.881667 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 05:24:47.881759 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 05:24:47.881973 (+ 214us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 05:24:47.881976 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 05:24:47.882075 (+ 99us) client_negotiation.cc:769] Sending connection context
0812 05:24:47.882155 (+ 80us) client_negotiation.cc:240] Negotiation successful
0812 05:24:47.882297 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"mutex_wait_us":152,"thread_start_us":91,"threads_started":1}
I20260812 05:24:47.883004 5360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:24:47.875062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43802 (local address 127.31.100.60:43459)
0812 05:24:47.875242 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0812 05:24:47.875246 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 05:24:47.875355 (+ 109us) server_negotiation.cc:407] Connection header received
0812 05:24:47.875380 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:24:47.875382 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:24:47.875405 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:24:47.875495 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:24:47.876039 (+ 544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:24:47.876830 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:24:47.877557 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:24:47.877849 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:24:47.877907 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:24:47.880134 (+ 2227us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 05:24:47.880143 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 05:24:47.880144 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 05:24:47.880151 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 05:24:47.881757 (+ 1606us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:24:47.882229 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:24:47.882232 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:24:47.882232 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:24:47.882287 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:24:47.882445 (+ 158us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:24:47.882446 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:24:47.882447 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:24:47.882600 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 05:24:47.882738 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0812 05:24:47.882833 (+ 95us) server_negotiation.cc:299] Negotiation successful
0812 05:24:47.882911 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152,"thread_start_us":89,"threads_started":1}
I20260812 05:24:47.883097 5357 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.100.61" port: 44651 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 05:24:47 dist-test-slave-42z9 krb5kdc[4978](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512287, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.100.60@KRBTEST.COM for kudu/127.31.100.61@KRBTEST.COM
Aug 12 05:24:47 dist-test-slave-42z9 krb5kdc[4978](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512287, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.100.62@KRBTEST.COM
I20260812 05:24:47.888877 5243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:24:47.883376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38674 (local address 127.31.100.61:44651)
0812 05:24:47.883418 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0812 05:24:47.883419 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0812 05:24:47.883426 (+ 7us) server_negotiation.cc:407] Connection header received
0812 05:24:47.883447 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:24:47.883448 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:24:47.883456 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:24:47.883501 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:24:47.883725 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:24:47.884237 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:24:47.884823 (+ 586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:24:47.885012 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:24:47.885058 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:24:47.886660 (+ 1602us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 05:24:47.886661 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 05:24:47.886662 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 05:24:47.886668 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0812 05:24:47.887942 (+ 1274us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:24:47.888231 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:24:47.888233 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:24:47.888233 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:24:47.888283 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:24:47.888438 (+ 155us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:24:47.888439 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:24:47.888439 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:24:47.888535 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 05:24:47.888586 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0812 05:24:47.888716 (+ 130us) server_negotiation.cc:299] Negotiation successful
0812 05:24:47.888790 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260812 05:24:47.888877 5359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:24:47.883300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.61:44651 (local address 127.0.0.1:38674)
0812 05:24:47.883338 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0812 05:24:47.883346 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0812 05:24:47.883376 (+ 30us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:24:47.883510 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:24:47.883511 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:24:47.883632 (+ 121us) client_negotiation.cc:189] Negotiated authn=SASL
0812 05:24:47.883659 (+ 27us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:24:47.883660 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:24:47.884300 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:24:47.884300 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:24:47.884752 (+ 452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:24:47.884754 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:24:47.885071 (+ 317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:24:47.885072 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:24:47.885128 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:24:47.885424 (+ 296us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 05:24:47.885428 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0812 05:24:47.886550 (+ 1122us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 05:24:47.888009 (+ 1459us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 05:24:47.888010 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 05:24:47.888011 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 05:24:47.888173 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 05:24:47.888334 (+ 161us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 05:24:47.888335 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 05:24:47.888335 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 05:24:47.888389 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 05:24:47.888587 (+ 198us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 05:24:47.888588 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 05:24:47.888627 (+ 39us) client_negotiation.cc:769] Sending connection context
0812 05:24:47.888687 (+ 60us) client_negotiation.cc:240] Negotiation successful
0812 05:24:47.888788 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27}
I20260812 05:24:47.889166 5357 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.100.60" port: 43459 } has no permanent_uuid. Determining permanent_uuid...
I20260812 05:24:47.890036 5115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:24:47.883750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44616 (local address 127.31.100.62:37187)
0812 05:24:47.883791 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0812 05:24:47.883793 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 05:24:47.883800 (+ 7us) server_negotiation.cc:407] Connection header received
0812 05:24:47.883830 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:24:47.883830 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:24:47.883839 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:24:47.883884 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:24:47.884252 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:24:47.884695 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:24:47.885480 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:24:47.885675 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:24:47.885724 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:24:47.887711 (+ 1987us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 05:24:47.887713 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 05:24:47.887713 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 05:24:47.887721 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0812 05:24:47.888943 (+ 1222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:24:47.889286 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:24:47.889287 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:24:47.889288 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:24:47.889340 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:24:47.889590 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:24:47.889591 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:24:47.889591 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:24:47.889692 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 05:24:47.889847 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0812 05:24:47.889881 (+ 34us) server_negotiation.cc:299] Negotiation successful
0812 05:24:47.889956 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
Aug 12 05:24:47 dist-test-slave-42z9 krb5kdc[4978](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512287, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.100.60@KRBTEST.COM for kudu/127.31.100.60@KRBTEST.COM
Aug 12 05:24:47 dist-test-slave-42z9 krb5kdc[4978](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512287, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.100.61@KRBTEST.COM
I20260812 05:24:47.894959 5359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:24:47.889370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.60:43459 (local address 127.0.0.1:43816)
0812 05:24:47.889396 (+ 26us) negotiation.cc:107] Waiting for socket to connect
0812 05:24:47.889405 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0812 05:24:47.889453 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:24:47.889541 (+ 88us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:24:47.889541 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:24:47.889658 (+ 117us) client_negotiation.cc:189] Negotiated authn=SASL
0812 05:24:47.889681 (+ 23us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:24:47.889683 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:24:47.890380 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:24:47.890381 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:24:47.890979 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:24:47.890981 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:24:47.891364 (+ 383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:24:47.891365 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:24:47.891418 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:24:47.891703 (+ 285us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 05:24:47.891708 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0812 05:24:47.892785 (+ 1077us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 05:24:47.894114 (+ 1329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 05:24:47.894115 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 05:24:47.894116 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 05:24:47.894261 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 05:24:47.894443 (+ 182us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 05:24:47.894443 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 05:24:47.894444 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 05:24:47.894485 (+ 41us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 05:24:47.894685 (+ 200us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 05:24:47.894686 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 05:24:47.894726 (+ 40us) client_negotiation.cc:769] Sending connection context
0812 05:24:47.894778 (+ 52us) client_negotiation.cc:240] Negotiation successful
0812 05:24:47.894871 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14}
I20260812 05:24:47.895020 5360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:24:47.889412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43816 (local address 127.31.100.60:43459)
0812 05:24:47.889464 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0812 05:24:47.889466 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 05:24:47.889472 (+ 6us) server_negotiation.cc:407] Connection header received
0812 05:24:47.889490 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:24:47.889490 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:24:47.889498 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:24:47.889540 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:24:47.889741 (+ 201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:24:47.890307 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:24:47.891050 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:24:47.891305 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:24:47.891351 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:24:47.892852 (+ 1501us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 05:24:47.892853 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 05:24:47.892853 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 05:24:47.892859 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0812 05:24:47.894030 (+ 1171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:24:47.894317 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:24:47.894318 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:24:47.894318 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:24:47.894369 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:24:47.894538 (+ 169us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:24:47.894538 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:24:47.894539 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:24:47.894633 (+ 94us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 05:24:47.894685 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0812 05:24:47.894805 (+ 120us) server_negotiation.cc:299] Negotiation successful
0812 05:24:47.894923 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":48,"server-negotiator.queue_time_us":18}
I20260812 05:24:47.896474 5357 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 66bc144962dc47d5aae05be95181bb55: Bootstrap starting.
I20260812 05:24:47.897329 5243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:24:47.890633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38678 (local address 127.31.100.61:44651)
0812 05:24:47.890668 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0812 05:24:47.890670 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 05:24:47.890677 (+ 7us) server_negotiation.cc:407] Connection header received
0812 05:24:47.890694 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:24:47.890695 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:24:47.890703 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:24:47.890758 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:24:47.891141 (+ 383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:24:47.891667 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:24:47.892373 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:24:47.892549 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:24:47.892600 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:24:47.894752 (+ 2152us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 05:24:47.894753 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 05:24:47.894753 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 05:24:47.894760 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 05:24:47.896206 (+ 1446us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:24:47.896525 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:24:47.896526 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:24:47.896527 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:24:47.896586 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:24:47.896746 (+ 160us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:24:47.896747 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:24:47.896747 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:24:47.896872 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 05:24:47.897035 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0812 05:24:47.897073 (+ 38us) server_negotiation.cc:299] Negotiation successful
0812 05:24:47.897247 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260812 05:24:47.897523 5357 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 66bc144962dc47d5aae05be95181bb55: Neither blocks nor log segments found. Creating new log.
I20260812 05:24:47.897836 5357 log.cc:824] T 00000000000000000000000000000000 P 66bc144962dc47d5aae05be95181bb55: Log is configured to *not* fsync() on all Append() calls
I20260812 05:24:47.898679 5357 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 66bc144962dc47d5aae05be95181bb55: No bootstrap required, opened a new log
I20260812 05:24:47.898687 32144 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 05:24:47.900188 5357 raft_consensus.cc:348] T 00000000000000000000000000000000 P 66bc144962dc47d5aae05be95181bb55 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e24c865796649b6bffc44f5c61d57f3" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 37187 } } peers { permanent_uuid: "5fc02b8d8ab74e6896097081da53ef7f" member_type: VOTER last_known_addr { host: "127.31.100.61" port: 44651 } } peers { permanent_uuid: "66bc144962dc47d5aae05be95181bb55" member_type: VOTER last_known_addr { host: "127.31.100.60" port: 43459 } }
I20260812 05:24:47.899683 5068 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "4e24c865796649b6bffc44f5c61d57f3"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44616
I20260812 05:24:47.900309 5357 raft_consensus.cc:729] T 00000000000000000000000000000000 P 66bc144962dc47d5aae05be95181bb55 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66bc144962dc47d5aae05be95181bb55, State: Initialized, Role: FOLLOWER
I20260812 05:24:47.900388 5357 consensus_queue.cc:260] T 00000000000000000000000000000000 P 66bc144962dc47d5aae05be95181bb55 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e24c865796649b6bffc44f5c61d57f3" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 37187 } } peers { permanent_uuid: "5fc02b8d8ab74e6896097081da53ef7f" member_type: VOTER last_known_addr { host: "127.31.100.61" port: 44651 } } peers { permanent_uuid: "66bc144962dc47d5aae05be95181bb55" member_type: VOTER last_known_addr { host: "127.31.100.60" port: 43459 } }
I20260812 05:24:47.900635 5370 sys_catalog.cc:455] T 00000000000000000000000000000000 P 66bc144962dc47d5aae05be95181bb55 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e24c865796649b6bffc44f5c61d57f3" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 37187 } } peers { permanent_uuid: "5fc02b8d8ab74e6896097081da53ef7f" member_type: VOTER last_known_addr { host: "127.31.100.61" port: 44651 } } peers { permanent_uuid: "66bc144962dc47d5aae05be95181bb55" member_type: VOTER last_known_addr { host: "127.31.100.60" port: 43459 } } }
I20260812 05:24:47.900701 5357 sys_catalog.cc:564] T 00000000000000000000000000000000 P 66bc144962dc47d5aae05be95181bb55 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 05:24:47.900728 5370 sys_catalog.cc:458] T 00000000000000000000000000000000 P 66bc144962dc47d5aae05be95181bb55 [sys.catalog]: This master's current role is: FOLLOWER
W20260812 05:24:47.902511 5384 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 66bc144962dc47d5aae05be95181bb55: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 05:24:47.902570 5384 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 05:24:47.902837 5248 master_runner.cc:186] Error getting master registration for 127.31.100.62:37187: OK, instance_id { permanent_uuid: "4e24c865796649b6bffc44f5c61d57f3" instance_seqno: 1786512287572408 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260812 05:24:47.905573 5248 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260812 05:24:47.914264 5108 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.100.60" port: 43459 } attempt: 2
Aug 12 05:24:47 dist-test-slave-42z9 krb5kdc[4978](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512287, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.100.62@KRBTEST.COM for kudu/127.31.100.60@KRBTEST.COM
I20260812 05:24:47.921523 5360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:24:47.914637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43830 (local address 127.31.100.60:43459)
0812 05:24:47.914732 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0812 05:24:47.914734 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 05:24:47.914742 (+ 8us) server_negotiation.cc:407] Connection header received
0812 05:24:47.914803 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:24:47.914805 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:24:47.914820 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:24:47.914902 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:24:47.915221 (+ 319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:24:47.915720 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:24:47.916389 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:24:47.916574 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:24:47.916627 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:24:47.918789 (+ 2162us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 05:24:47.918791 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 05:24:47.918791 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 05:24:47.918800 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0812 05:24:47.920297 (+ 1497us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:24:47.920675 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:24:47.920676 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:24:47.920677 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:24:47.920729 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:24:47.920902 (+ 173us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:24:47.920903 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:24:47.920904 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:24:47.921088 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 05:24:47.921151 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0812 05:24:47.921337 (+ 186us) server_negotiation.cc:299] Negotiation successful
0812 05:24:47.921412 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260812 05:24:47.921541 5114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:24:47.914571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.60:43459 (local address 127.0.0.1:43830)
0812 05:24:47.914625 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0812 05:24:47.914636 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0812 05:24:47.914681 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:24:47.914931 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:24:47.914933 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:24:47.915104 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0812 05:24:47.915147 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:24:47.915149 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:24:47.915796 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:24:47.915797 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:24:47.916311 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:24:47.916312 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:24:47.916631 (+ 319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:24:47.916631 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:24:47.916681 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:24:47.917152 (+ 471us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 05:24:47.917158 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0812 05:24:47.918700 (+ 1542us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 05:24:47.920377 (+ 1677us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 05:24:47.920379 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 05:24:47.920379 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 05:24:47.920600 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 05:24:47.920784 (+ 184us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 05:24:47.920785 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 05:24:47.920785 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 05:24:47.920851 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 05:24:47.921171 (+ 320us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 05:24:47.921172 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 05:24:47.921226 (+ 54us) client_negotiation.cc:769] Sending connection context
0812 05:24:47.921311 (+ 85us) client_negotiation.cc:240] Negotiation successful
0812 05:24:47.921405 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260812 05:24:47.923153 5108 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4e24c865796649b6bffc44f5c61d57f3: Bootstrap starting.
I20260812 05:24:47.923877 5108 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4e24c865796649b6bffc44f5c61d57f3: Neither blocks nor log segments found. Creating new log.
I20260812 05:24:47.924185 5108 log.cc:824] T 00000000000000000000000000000000 P 4e24c865796649b6bffc44f5c61d57f3: Log is configured to *not* fsync() on all Append() calls
I20260812 05:24:47.924998 5108 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4e24c865796649b6bffc44f5c61d57f3: No bootstrap required, opened a new log
I20260812 05:24:47.926363 5108 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4e24c865796649b6bffc44f5c61d57f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e24c865796649b6bffc44f5c61d57f3" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 37187 } } peers { permanent_uuid: "5fc02b8d8ab74e6896097081da53ef7f" member_type: VOTER last_known_addr { host: "127.31.100.61" port: 44651 } } peers { permanent_uuid: "66bc144962dc47d5aae05be95181bb55" member_type: VOTER last_known_addr { host: "127.31.100.60" port: 43459 } }
I20260812 05:24:47.926508 5108 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4e24c865796649b6bffc44f5c61d57f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e24c865796649b6bffc44f5c61d57f3, State: Initialized, Role: FOLLOWER
I20260812 05:24:47.926591 5108 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4e24c865796649b6bffc44f5c61d57f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e24c865796649b6bffc44f5c61d57f3" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 37187 } } peers { permanent_uuid: "5fc02b8d8ab74e6896097081da53ef7f" member_type: VOTER last_known_addr { host: "127.31.100.61" port: 44651 } } peers { permanent_uuid: "66bc144962dc47d5aae05be95181bb55" member_type: VOTER last_known_addr { host: "127.31.100.60" port: 43459 } }
I20260812 05:24:47.926896 5108 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4e24c865796649b6bffc44f5c61d57f3 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 05:24:47.926862 5390 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e24c865796649b6bffc44f5c61d57f3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e24c865796649b6bffc44f5c61d57f3" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 37187 } } peers { permanent_uuid: "5fc02b8d8ab74e6896097081da53ef7f" member_type: VOTER last_known_addr { host: "127.31.100.61" port: 44651 } } peers { permanent_uuid: "66bc144962dc47d5aae05be95181bb55" member_type: VOTER last_known_addr { host: "127.31.100.60" port: 43459 } } }
I20260812 05:24:47.926959 5390 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e24c865796649b6bffc44f5c61d57f3 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 05:24:47.927043 5068 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4e24c865796649b6bffc44f5c61d57f3 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260812 05:24:47.927105 5068 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4e24c865796649b6bffc44f5c61d57f3 [term 0 FOLLOWER]: Advancing to term 1
I20260812 05:24:47.927770 5068 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4e24c865796649b6bffc44f5c61d57f3 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e24c865796649b6bffc44f5c61d57f3" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 37187 } } peers { permanent_uuid: "5fc02b8d8ab74e6896097081da53ef7f" member_type: VOTER last_known_addr { host: "127.31.100.61" port: 44651 } } peers { permanent_uuid: "66bc144962dc47d5aae05be95181bb55" member_type: VOTER last_known_addr { host: "127.31.100.60" port: 43459 } }
I20260812 05:24:47.928229 5315 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4e24c865796649b6bffc44f5c61d57f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "66bc144962dc47d5aae05be95181bb55"
I20260812 05:24:47.928356 5315 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 66bc144962dc47d5aae05be95181bb55 [term 0 FOLLOWER]: Advancing to term 1
W20260812 05:24:47.929147 5408 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4e24c865796649b6bffc44f5c61d57f3: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 05:24:47.929255 5408 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 05:24:47.929677 5315 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 66bc144962dc47d5aae05be95181bb55 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e24c865796649b6bffc44f5c61d57f3 in term 1.
I20260812 05:24:47.930066 5068 leader_election.cc:288] T 00000000000000000000000000000000 P 4e24c865796649b6bffc44f5c61d57f3 [CANDIDATE]: Term 1 election: Requested vote from peers 5fc02b8d8ab74e6896097081da53ef7f (127.31.100.61:44651), 66bc144962dc47d5aae05be95181bb55 (127.31.100.60:43459)
I20260812 05:24:47.930086 5037 leader_election.cc:302] T 00000000000000000000000000000000 P 4e24c865796649b6bffc44f5c61d57f3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e24c865796649b6bffc44f5c61d57f3, 66bc144962dc47d5aae05be95181bb55; no voters:
I20260812 05:24:47.930219 5390 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4e24c865796649b6bffc44f5c61d57f3 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 05:24:47.930352 5390 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4e24c865796649b6bffc44f5c61d57f3 [term 1 LEADER]: Becoming Leader. State: Replica: 4e24c865796649b6bffc44f5c61d57f3, State: Running, Role: LEADER
I20260812 05:24:47.930454 5390 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4e24c865796649b6bffc44f5c61d57f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e24c865796649b6bffc44f5c61d57f3" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 37187 } } peers { permanent_uuid: "5fc02b8d8ab74e6896097081da53ef7f" member_type: VOTER last_known_addr { host: "127.31.100.61" port: 44651 } } peers { permanent_uuid: "66bc144962dc47d5aae05be95181bb55" member_type: VOTER last_known_addr { host: "127.31.100.60" port: 43459 } }
I20260812 05:24:47.930572 4997 master_runner.cc:186] Error getting master registration for 127.31.100.61:44651: OK, instance_id { permanent_uuid: "5fc02b8d8ab74e6896097081da53ef7f" instance_seqno: 1786512287720653 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260812 05:24:47.932902 4997 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260812 05:24:47.933126 5411 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e24c865796649b6bffc44f5c61d57f3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4e24c865796649b6bffc44f5c61d57f3. Latest consensus state: current_term: 1 leader_uuid: "4e24c865796649b6bffc44f5c61d57f3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e24c865796649b6bffc44f5c61d57f3" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 37187 } } peers { permanent_uuid: "5fc02b8d8ab74e6896097081da53ef7f" member_type: VOTER last_known_addr { host: "127.31.100.61" port: 44651 } } peers { permanent_uuid: "66bc144962dc47d5aae05be95181bb55" member_type: VOTER last_known_addr { host: "127.31.100.60" port: 43459 } } }
I20260812 05:24:47.933306 5411 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e24c865796649b6bffc44f5c61d57f3 [sys.catalog]: This master's current role is: LEADER
I20260812 05:24:47.937651 5429 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 05:24:47.937866 5429 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 05:24:47.940322 5360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:24:47.931892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43840 (local address 127.31.100.60:43459)
0812 05:24:47.932978 (+ 1086us) server_negotiation.cc:206] Beginning negotiation
0812 05:24:47.932980 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 05:24:47.932988 (+ 8us) server_negotiation.cc:407] Connection header received
0812 05:24:47.933006 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:24:47.933007 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:24:47.933016 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:24:47.933067 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:24:47.934043 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:24:47.934700 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:24:47.935682 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:24:47.935966 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:24:47.936009 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:24:47.937494 (+ 1485us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 05:24:47.937495 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 05:24:47.937495 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 05:24:47.937502 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 05:24:47.939109 (+ 1607us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:24:47.939524 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:24:47.939526 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:24:47.939526 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:24:47.939583 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:24:47.939776 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:24:47.939777 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:24:47.939778 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:24:47.939895 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 05:24:47.939973 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0812 05:24:47.940163 (+ 190us) server_negotiation.cc:299] Negotiation successful
0812 05:24:47.940234 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1065}
I20260812 05:24:47.946478 5315 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 66bc144962dc47d5aae05be95181bb55 [term 1 FOLLOWER]: Refusing update from remote peer 4e24c865796649b6bffc44f5c61d57f3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 05:24:47.946791 5411 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4e24c865796649b6bffc44f5c61d57f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66bc144962dc47d5aae05be95181bb55" member_type: VOTER last_known_addr { host: "127.31.100.60" port: 43459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 05:24:47.947335 5243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:24:47.932524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38680 (local address 127.31.100.61:44651)
0812 05:24:47.932577 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0812 05:24:47.932579 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 05:24:47.934403 (+ 1824us) server_negotiation.cc:407] Connection header received
0812 05:24:47.934657 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:24:47.934658 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:24:47.934668 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:24:47.934706 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:24:47.937532 (+ 2826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:24:47.938220 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:24:47.940931 (+ 2711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:24:47.941562 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:24:47.941609 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:24:47.943856 (+ 2247us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 05:24:47.943857 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 05:24:47.943858 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 05:24:47.943865 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 05:24:47.945502 (+ 1637us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:24:47.946017 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:24:47.946019 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:24:47.946019 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:24:47.946091 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:24:47.946392 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:24:47.946393 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:24:47.946393 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:24:47.946531 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 05:24:47.946614 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0812 05:24:47.947165 (+ 551us) server_negotiation.cc:299] Negotiation successful
0812 05:24:47.947251 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260812 05:24:47.948143 5370 sys_catalog.cc:455] T 00000000000000000000000000000000 P 66bc144962dc47d5aae05be95181bb55 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4e24c865796649b6bffc44f5c61d57f3. Latest consensus state: current_term: 1 leader_uuid: "4e24c865796649b6bffc44f5c61d57f3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e24c865796649b6bffc44f5c61d57f3" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 37187 } } peers { permanent_uuid: "5fc02b8d8ab74e6896097081da53ef7f" member_type: VOTER last_known_addr { host: "127.31.100.61" port: 44651 } } peers { permanent_uuid: "66bc144962dc47d5aae05be95181bb55" member_type: VOTER last_known_addr { host: "127.31.100.60" port: 43459 } } }
I20260812 05:24:47.948227 5370 sys_catalog.cc:458] T 00000000000000000000000000000000 P 66bc144962dc47d5aae05be95181bb55 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 05:24:47.949339 5115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:24:47.932228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44620 (local address 127.31.100.62:37187)
0812 05:24:47.932265 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0812 05:24:47.932267 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 05:24:47.934466 (+ 2199us) server_negotiation.cc:407] Connection header received
0812 05:24:47.934778 (+ 312us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:24:47.934778 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:24:47.934801 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:24:47.934828 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:24:47.936664 (+ 1836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:24:47.937374 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:24:47.941760 (+ 4386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:24:47.942044 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:24:47.942106 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:24:47.943516 (+ 1410us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 05:24:47.943517 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 05:24:47.943518 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 05:24:47.943525 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 05:24:47.946177 (+ 2652us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:24:47.948576 (+ 2399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:24:47.948577 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:24:47.948578 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:24:47.948639 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:24:47.948842 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:24:47.948843 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:24:47.948843 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:24:47.948971 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 05:24:47.949081 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0812 05:24:47.949192 (+ 111us) server_negotiation.cc:299] Negotiation successful
0812 05:24:47.949260 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260812 05:24:47.950093 5390 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e24c865796649b6bffc44f5c61d57f3 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4e24c865796649b6bffc44f5c61d57f3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e24c865796649b6bffc44f5c61d57f3" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 37187 } } peers { permanent_uuid: "5fc02b8d8ab74e6896097081da53ef7f" member_type: VOTER last_known_addr { host: "127.31.100.61" port: 44651 } } peers { permanent_uuid: "66bc144962dc47d5aae05be95181bb55" member_type: VOTER last_known_addr { host: "127.31.100.60" port: 43459 } } }
I20260812 05:24:47.950176 5390 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e24c865796649b6bffc44f5c61d57f3 [sys.catalog]: This master's current role is: LEADER
I20260812 05:24:47.950165 5370 sys_catalog.cc:455] T 00000000000000000000000000000000 P 66bc144962dc47d5aae05be95181bb55 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4e24c865796649b6bffc44f5c61d57f3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e24c865796649b6bffc44f5c61d57f3" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 37187 } } peers { permanent_uuid: "5fc02b8d8ab74e6896097081da53ef7f" member_type: VOTER last_known_addr { host: "127.31.100.61" port: 44651 } } peers { permanent_uuid: "66bc144962dc47d5aae05be95181bb55" member_type: VOTER last_known_addr { host: "127.31.100.60" port: 43459 } } }
I20260812 05:24:47.950227 5370 sys_catalog.cc:458] T 00000000000000000000000000000000 P 66bc144962dc47d5aae05be95181bb55 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 05:24:47.950785 5431 mvcc.cc:205] Tried to move back new op lower bound from 7317554331425693696 to 7317554331363729408. Current Snapshot: MvccSnapshot[applied={T|T < 7317554331425693696}]
I20260812 05:24:47.951924 5237 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.100.60" port: 43459 } attempt: 3
Aug 12 05:24:47 dist-test-slave-42z9 krb5kdc[4978](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512287, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.100.61@KRBTEST.COM for kudu/127.31.100.60@KRBTEST.COM
I20260812 05:24:47.957829 5429 catalog_manager.cc:1347] Generated new cluster ID: 93ab46f43cc24693ae86b0fd9f09adbc
I20260812 05:24:47.957885 5429 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 05:24:47.959103 5360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:24:47.952214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43852 (local address 127.31.100.60:43459)
0812 05:24:47.952252 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0812 05:24:47.952253 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0812 05:24:47.952260 (+ 7us) server_negotiation.cc:407] Connection header received
0812 05:24:47.952314 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:24:47.952315 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:24:47.952324 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:24:47.952369 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:24:47.952675 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:24:47.953264 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:24:47.954202 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:24:47.954397 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:24:47.954449 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:24:47.956532 (+ 2083us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 05:24:47.956533 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 05:24:47.956533 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 05:24:47.956539 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0812 05:24:47.958085 (+ 1546us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:24:47.958399 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:24:47.958413 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:24:47.958413 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:24:47.958464 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:24:47.958623 (+ 159us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:24:47.958624 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:24:47.958624 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:24:47.958757 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 05:24:47.958808 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0812 05:24:47.958964 (+ 156us) server_negotiation.cc:299] Negotiation successful
0812 05:24:47.959025 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260812 05:24:47.959151 5241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:24:47.952173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.60:43459 (local address 127.0.0.1:43852)
0812 05:24:47.952217 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0812 05:24:47.952226 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0812 05:24:47.952262 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:24:47.952415 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:24:47.952416 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:24:47.952567 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
0812 05:24:47.952608 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:24:47.952609 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:24:47.953328 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:24:47.953329 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:24:47.954128 (+ 799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:24:47.954132 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:24:47.954458 (+ 326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:24:47.954459 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:24:47.954510 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:24:47.954891 (+ 381us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 05:24:47.954896 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0812 05:24:47.956453 (+ 1557us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 05:24:47.958149 (+ 1696us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 05:24:47.958150 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 05:24:47.958150 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 05:24:47.958346 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 05:24:47.958509 (+ 163us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 05:24:47.958510 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 05:24:47.958511 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 05:24:47.958579 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 05:24:47.958802 (+ 223us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 05:24:47.958803 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 05:24:47.958859 (+ 56us) client_negotiation.cc:769] Sending connection context
0812 05:24:47.958925 (+ 66us) client_negotiation.cc:240] Negotiation successful
0812 05:24:47.959013 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260812 05:24:47.960503 5237 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5fc02b8d8ab74e6896097081da53ef7f: Bootstrap starting.
I20260812 05:24:47.961090 5237 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5fc02b8d8ab74e6896097081da53ef7f: Neither blocks nor log segments found. Creating new log.
I20260812 05:24:47.961346 5237 log.cc:824] T 00000000000000000000000000000000 P 5fc02b8d8ab74e6896097081da53ef7f: Log is configured to *not* fsync() on all Append() calls
I20260812 05:24:47.962167 5237 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5fc02b8d8ab74e6896097081da53ef7f: No bootstrap required, opened a new log
I20260812 05:24:47.964087 5237 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5fc02b8d8ab74e6896097081da53ef7f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e24c865796649b6bffc44f5c61d57f3" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 37187 } } peers { permanent_uuid: "5fc02b8d8ab74e6896097081da53ef7f" member_type: VOTER last_known_addr { host: "127.31.100.61" port: 44651 } } peers { permanent_uuid: "66bc144962dc47d5aae05be95181bb55" member_type: VOTER last_known_addr { host: "127.31.100.60" port: 43459 } }
I20260812 05:24:47.964218 5237 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5fc02b8d8ab74e6896097081da53ef7f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fc02b8d8ab74e6896097081da53ef7f, State: Initialized, Role: FOLLOWER
I20260812 05:24:47.964295 5237 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5fc02b8d8ab74e6896097081da53ef7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e24c865796649b6bffc44f5c61d57f3" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 37187 } } peers { permanent_uuid: "5fc02b8d8ab74e6896097081da53ef7f" member_type: VOTER last_known_addr { host: "127.31.100.61" port: 44651 } } peers { permanent_uuid: "66bc144962dc47d5aae05be95181bb55" member_type: VOTER last_known_addr { host: "127.31.100.60" port: 43459 } }
I20260812 05:24:47.964612 5237 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5fc02b8d8ab74e6896097081da53ef7f [sys.catalog]: configured and running, proceeding with master startup.
I20260812 05:24:47.964679 5191 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5fc02b8d8ab74e6896097081da53ef7f [term 0 FOLLOWER]: Advancing to term 1
I20260812 05:24:47.965852 5193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4e24c865796649b6bffc44f5c61d57f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "5fc02b8d8ab74e6896097081da53ef7f"
I20260812 05:24:47.965937 5191 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 5fc02b8d8ab74e6896097081da53ef7f [term 1 FOLLOWER]: Refusing update from remote peer 4e24c865796649b6bffc44f5c61d57f3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 05:24:47.966938 5193 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 5fc02b8d8ab74e6896097081da53ef7f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e24c865796649b6bffc44f5c61d57f3 in term 1.
I20260812 05:24:47.967140 5390 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4e24c865796649b6bffc44f5c61d57f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5fc02b8d8ab74e6896097081da53ef7f" member_type: VOTER last_known_addr { host: "127.31.100.61" port: 44651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260812 05:24:47.968102 5462 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5fc02b8d8ab74e6896097081da53ef7f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 05:24:47.968159 5462 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260812 05:24:47.968537 5164 tablet.cc:2369] T 00000000000000000000000000000000 P 5fc02b8d8ab74e6896097081da53ef7f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 05:24:47.964577 5438 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5fc02b8d8ab74e6896097081da53ef7f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e24c865796649b6bffc44f5c61d57f3" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 37187 } } peers { permanent_uuid: "5fc02b8d8ab74e6896097081da53ef7f" member_type: VOTER last_known_addr { host: "127.31.100.61" port: 44651 } } peers { permanent_uuid: "66bc144962dc47d5aae05be95181bb55" member_type: VOTER last_known_addr { host: "127.31.100.60" port: 43459 } } }
I20260812 05:24:47.970207 5438 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5fc02b8d8ab74e6896097081da53ef7f [sys.catalog]: This master's current role is: FOLLOWER
I20260812 05:24:47.970304 5438 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5fc02b8d8ab74e6896097081da53ef7f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4e24c865796649b6bffc44f5c61d57f3. Latest consensus state: current_term: 1 leader_uuid: "4e24c865796649b6bffc44f5c61d57f3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e24c865796649b6bffc44f5c61d57f3" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 37187 } } peers { permanent_uuid: "5fc02b8d8ab74e6896097081da53ef7f" member_type: VOTER last_known_addr { host: "127.31.100.61" port: 44651 } } peers { permanent_uuid: "66bc144962dc47d5aae05be95181bb55" member_type: VOTER last_known_addr { host: "127.31.100.60" port: 43459 } } }
I20260812 05:24:47.970404 5438 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5fc02b8d8ab74e6896097081da53ef7f [sys.catalog]: This master's current role is: FOLLOWER
I20260812 05:24:47.970476 5438 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5fc02b8d8ab74e6896097081da53ef7f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4e24c865796649b6bffc44f5c61d57f3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e24c865796649b6bffc44f5c61d57f3" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 37187 } } peers { permanent_uuid: "5fc02b8d8ab74e6896097081da53ef7f" member_type: VOTER last_known_addr { host: "127.31.100.61" port: 44651 } } peers { permanent_uuid: "66bc144962dc47d5aae05be95181bb55" member_type: VOTER last_known_addr { host: "127.31.100.60" port: 43459 } } }
I20260812 05:24:47.970564 5438 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5fc02b8d8ab74e6896097081da53ef7f [sys.catalog]: This master's current role is: FOLLOWER
I20260812 05:24:47.977861 5466 mvcc.cc:205] Tried to move back new op lower bound from 7317554331425693696 to 7317554331363729408. Current Snapshot: MvccSnapshot[applied={T|T < 7317554331425693696}]
I20260812 05:24:47.979657 5411 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e24c865796649b6bffc44f5c61d57f3 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4e24c865796649b6bffc44f5c61d57f3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e24c865796649b6bffc44f5c61d57f3" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 37187 } } peers { permanent_uuid: "5fc02b8d8ab74e6896097081da53ef7f" member_type: VOTER last_known_addr { host: "127.31.100.61" port: 44651 } } peers { permanent_uuid: "66bc144962dc47d5aae05be95181bb55" member_type: VOTER last_known_addr { host: "127.31.100.60" port: 43459 } } }
I20260812 05:24:47.979755 5411 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e24c865796649b6bffc44f5c61d57f3 [sys.catalog]: This master's current role is: LEADER
I20260812 05:24:47.981904 5429 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 05:24:47.982566 5429 catalog_manager.cc:1504] Loading token signing keys...
I20260812 05:24:48.005935 5429 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4e24c865796649b6bffc44f5c61d57f3: Generated new TSK 0
I20260812 05:24:48.006143 5429 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260812 05:24:48.032377 5360 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 05:24:48.031326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43856 (local address 127.31.100.60:43459)
0812 05:24:48.031370 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0812 05:24:48.031372 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 05:24:48.031712 (+ 340us) server_negotiation.cc:407] Connection header received
0812 05:24:48.031816 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:24:48.031817 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:24:48.031828 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:24:48.031858 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:24:48.032309 (+ 451us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43856: BlockingRecv error: recv got EOF from 127.0.0.1:43856 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260812 05:24:48.033629 5115 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 05:24:48.032724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44622 (local address 127.31.100.62:37187)
0812 05:24:48.032761 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0812 05:24:48.032763 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 05:24:48.033083 (+ 320us) server_negotiation.cc:407] Connection header received
0812 05:24:48.033175 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:24:48.033176 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:24:48.033186 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:24:48.033215 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:24:48.033554 (+ 339us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44622: BlockingRecv error: recv got EOF from 127.0.0.1:44622 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260812 05:24:48.034407 5243 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 05:24:48.032965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38682 (local address 127.31.100.61:44651)
0812 05:24:48.033003 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0812 05:24:48.033005 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 05:24:48.033894 (+ 889us) server_negotiation.cc:407] Connection header received
0812 05:24:48.033988 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:24:48.033988 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:24:48.033999 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:24:48.034028 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:24:48.034348 (+ 320us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38682: BlockingRecv error: recv got EOF from 127.0.0.1:38682 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260812 05:24:48.036567 5243 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 05:24:48.035857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38686 (local address 127.31.100.61:44651)
0812 05:24:48.035889 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0812 05:24:48.035891 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 05:24:48.036065 (+ 174us) server_negotiation.cc:407] Connection header received
0812 05:24:48.036167 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:24:48.036168 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:24:48.036189 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:24:48.036217 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:24:48.036525 (+ 308us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38686: BlockingRecv error: recv got EOF from 127.0.0.1:38686 (error 108)
Metrics: {"server-negotiator.queue_time_us":15}
W20260812 05:24:48.039537 5243 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 05:24:48.038797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38698 (local address 127.31.100.61:44651)
0812 05:24:48.038829 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0812 05:24:48.038831 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 05:24:48.039039 (+ 208us) server_negotiation.cc:407] Connection header received
0812 05:24:48.039128 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:24:48.039129 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:24:48.039139 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:24:48.039166 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:24:48.039496 (+ 330us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38698: BlockingRecv error: recv got EOF from 127.0.0.1:38698 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260812 05:24:48.043354 5243 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 05:24:48.042649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38706 (local address 127.31.100.61:44651)
0812 05:24:48.042682 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0812 05:24:48.042684 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 05:24:48.042890 (+ 206us) server_negotiation.cc:407] Connection header received
0812 05:24:48.042978 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:24:48.042979 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:24:48.042988 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:24:48.043015 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:24:48.043315 (+ 300us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38706: BlockingRecv error: recv got EOF from 127.0.0.1:38706 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260812 05:24:48.049357 5243 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 05:24:48.048480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38708 (local address 127.31.100.61:44651)
0812 05:24:48.048515 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0812 05:24:48.048517 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 05:24:48.048772 (+ 255us) server_negotiation.cc:407] Connection header received
0812 05:24:48.048865 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:24:48.048866 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:24:48.048875 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:24:48.048903 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:24:48.049314 (+ 411us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38708: BlockingRecv error: recv got EOF from 127.0.0.1:38708 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260812 05:24:48.059507 5243 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 05:24:48.058667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38718 (local address 127.31.100.61:44651)
0812 05:24:48.058705 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0812 05:24:48.058707 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 05:24:48.058931 (+ 224us) server_negotiation.cc:407] Connection header received
0812 05:24:48.059022 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:24:48.059023 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:24:48.059034 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:24:48.059064 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:24:48.059455 (+ 391us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38718: BlockingRecv error: recv got EOF from 127.0.0.1:38718 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260812 05:24:48.077884 5243 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 05:24:48.076876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38722 (local address 127.31.100.61:44651)
0812 05:24:48.076922 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0812 05:24:48.076925 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 05:24:48.077192 (+ 267us) server_negotiation.cc:407] Connection header received
0812 05:24:48.077298 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:24:48.077299 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:24:48.077310 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:24:48.077340 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:24:48.077825 (+ 485us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38722: BlockingRecv error: recv got EOF from 127.0.0.1:38722 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
W20260812 05:24:48.111946 5243 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 05:24:48.111165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38738 (local address 127.31.100.61:44651)
0812 05:24:48.111230 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0812 05:24:48.111232 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 05:24:48.111361 (+ 129us) server_negotiation.cc:407] Connection header received
0812 05:24:48.111446 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:24:48.111447 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:24:48.111458 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:24:48.111508 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:24:48.111883 (+ 375us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38738: BlockingRecv error: recv got EOF from 127.0.0.1:38738 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
W20260812 05:24:48.177871 5243 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 05:24:48.177149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38744 (local address 127.31.100.61:44651)
0812 05:24:48.177231 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0812 05:24:48.177234 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 05:24:48.177291 (+ 57us) server_negotiation.cc:407] Connection header received
0812 05:24:48.177369 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:24:48.177371 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:24:48.177383 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:24:48.177434 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:24:48.177805 (+ 371us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38744: BlockingRecv error: recv got EOF from 127.0.0.1:38744 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
W20260812 05:24:48.308027 5243 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 05:24:48.307349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38750 (local address 127.31.100.61:44651)
0812 05:24:48.307419 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0812 05:24:48.307421 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 05:24:48.307508 (+ 87us) server_negotiation.cc:407] Connection header received
0812 05:24:48.307579 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:24:48.307580 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:24:48.307592 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:24:48.307644 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:24:48.307962 (+ 318us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38750: BlockingRecv error: recv got EOF from 127.0.0.1:38750 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
W20260812 05:24:48.565779 5243 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 05:24:48.565144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38760 (local address 127.31.100.61:44651)
0812 05:24:48.565204 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0812 05:24:48.565206 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 05:24:48.565216 (+ 10us) server_negotiation.cc:407] Connection header received
0812 05:24:48.565280 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:24:48.565281 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:24:48.565293 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:24:48.565366 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:24:48.565714 (+ 348us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38760: BlockingRecv error: recv got EOF from 127.0.0.1:38760 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
I20260812 05:24:48.902987 5384 catalog_manager.cc:1259] Loaded cluster ID: 93ab46f43cc24693ae86b0fd9f09adbc
I20260812 05:24:48.903070 5384 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 66bc144962dc47d5aae05be95181bb55: loading cluster ID for follower catalog manager: success
I20260812 05:24:48.903861 5384 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 66bc144962dc47d5aae05be95181bb55: acquiring CA information for follower catalog manager: success
I20260812 05:24:48.904191 5384 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 66bc144962dc47d5aae05be95181bb55: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260812 05:24:48.968578 5462 catalog_manager.cc:1259] Loaded cluster ID: 93ab46f43cc24693ae86b0fd9f09adbc
I20260812 05:24:48.968654 5462 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 5fc02b8d8ab74e6896097081da53ef7f: loading cluster ID for follower catalog manager: success
I20260812 05:24:48.969462 5462 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 5fc02b8d8ab74e6896097081da53ef7f: acquiring CA information for follower catalog manager: success
I20260812 05:24:48.969753 5462 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 5fc02b8d8ab74e6896097081da53ef7f: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260812 05:24:49.081387 5577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:24:49.078968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38762 (local address 127.31.100.61:44651)
0812 05:24:49.079080 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0812 05:24:49.079082 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 05:24:49.079091 (+ 9us) server_negotiation.cc:407] Connection header received
0812 05:24:49.079123 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:24:49.079124 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:24:49.079174 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:24:49.079246 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 05:24:49.079487 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:24:49.079981 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:24:49.080743 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:24:49.080924 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:24:49.080966 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:24:49.081113 (+ 147us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 05:24:49.081220 (+ 107us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 05:24:49.081274 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0812 05:24:49.081289 (+ 15us) server_negotiation.cc:299] Negotiation successful
0812 05:24:49.081305 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94,"thread_start_us":64,"threads_started":1}
I20260812 05:24:49.085482 5585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:24:49.083170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43860 (local address 127.31.100.60:43459)
0812 05:24:49.083285 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0812 05:24:49.083287 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 05:24:49.083296 (+ 9us) server_negotiation.cc:407] Connection header received
0812 05:24:49.083338 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:24:49.083339 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:24:49.083376 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:24:49.083442 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 05:24:49.083541 (+ 99us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:24:49.084031 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:24:49.084749 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:24:49.084945 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:24:49.084999 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:24:49.085141 (+ 142us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 05:24:49.085245 (+ 104us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 05:24:49.085310 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0812 05:24:49.085348 (+ 38us) server_negotiation.cc:299] Negotiation successful
0812 05:24:49.085369 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93,"thread_start_us":50,"threads_started":1}
I20260812 05:24:49.089457 5585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:24:49.086992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43866 (local address 127.31.100.60:43459)
0812 05:24:49.087030 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0812 05:24:49.087031 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0812 05:24:49.087080 (+ 49us) server_negotiation.cc:407] Connection header received
0812 05:24:49.087135 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:24:49.087136 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:24:49.087147 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:24:49.087203 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 05:24:49.087315 (+ 112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:24:49.087936 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:24:49.088822 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:24:49.088998 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:24:49.089053 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:24:49.089192 (+ 139us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 05:24:49.089220 (+ 28us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 05:24:49.089271 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0812 05:24:49.089338 (+ 67us) server_negotiation.cc:299] Negotiation successful
0812 05:24:49.089368 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260812 05:24:49.090169 5597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:24:49.087044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44624 (local address 127.31.100.62:37187)
0812 05:24:49.087148 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0812 05:24:49.087150 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 05:24:49.087257 (+ 107us) server_negotiation.cc:407] Connection header received
0812 05:24:49.087310 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:24:49.087312 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:24:49.087365 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:24:49.087451 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 05:24:49.087553 (+ 102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:24:49.088350 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:24:49.089205 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:24:49.089528 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:24:49.089591 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:24:49.089840 (+ 249us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 05:24:49.089948 (+ 108us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 05:24:49.090016 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0812 05:24:49.090059 (+ 43us) server_negotiation.cc:299] Negotiation successful
0812 05:24:49.090085 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85,"thread_start_us":48,"threads_started":1}
I20260812 05:24:49.090746 5577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:24:49.087185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38766 (local address 127.31.100.61:44651)
0812 05:24:49.087221 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0812 05:24:49.087223 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 05:24:49.087330 (+ 107us) server_negotiation.cc:407] Connection header received
0812 05:24:49.087422 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:24:49.087423 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:24:49.087435 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:24:49.087481 (+ 46us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 05:24:49.087634 (+ 153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:24:49.088364 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:24:49.090086 (+ 1722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:24:49.090325 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:24:49.090387 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:24:49.090505 (+ 118us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 05:24:49.090536 (+ 31us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 05:24:49.090575 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0812 05:24:49.090654 (+ 79us) server_negotiation.cc:299] Negotiation successful
0812 05:24:49.090678 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260812 05:24:49.091814 32144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0waFvT/build/release/bin/kudu with pid 4997
I20260812 05:24:49.105973 32144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0waFvT/build/release/bin/kudu with pid 5122
I20260812 05:24:49.116696 32144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0waFvT/build/release/bin/kudu with pid 5248
2026-08-12T05:24:49Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (2982 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786512017613366-32144-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 05:24:49 dist-test-slave-42z9 krb5kdc[5602](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 05:24:49 dist-test-slave-42z9 krb5kdc[5602](info): set up 2 sockets
Aug 12 05:24:49 dist-test-slave-42z9 krb5kdc[5602](info): commencing operation
krb5kdc: starting...
W20260812 05:24:50.299811 32144 mini_kdc.cc:121] Time spent starting KDC: real 1.159s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786512017613366-32144-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786512017613366-32144-0/krb5kdc/test-user.keytab.
Aug 12 05:24:50 dist-test-slave-42z9 krb5kdc[5602](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512290, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T05:24:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T05:24:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.100.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.100.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.100.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.62.keytab.
Entry for principal kudu/127.31.100.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.62.keytab.
I20260812 05:24:50.413578 32144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0waFvT/build/release/bin/kudu
/tmp/dist-test-task0waFvT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786512017613366-32144-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786512017613366-32144-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786512017613366-32144-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786512017613366-32144-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.100.62:38737
--webserver_interface=127.31.100.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.62.keytab
--principal=kudu/127.31.100.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.100.20:40157
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.100.62:38737,127.31.100.61:42987,127.31.100.60:44447
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786512017613366-32144-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786512017613366-32144-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786512017613366-32144-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 05:24:50.494405 5623 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260812 05:24:50.494616 5623 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 05:24:50.494632 5623 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 05:24:50.494654 5623 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 05:24:50.495782 5623 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 05:24:50.495823 5623 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 05:24:50.495836 5623 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 05:24:50.495859 5623 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 05:24:50.495882 5623 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 05:24:50.497362 5623 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.100.20:40157
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786512017613366-32144-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786512017613366-32144-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.100.62:38737,127.31.100.61:42987,127.31.100.60:44447
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.62.keytab
--principal=kudu/127.31.100.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.100.62:38737
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786512017613366-32144-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.100.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5623
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786512017613366-32144-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 04:20:34 UTC on ccycloud.quasar-phqdqp.root.comops.site
build id 4581871
I20260812 05:24:50.497608 5623 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 05:24:50.497793 5623 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 05:24:50.502321 5634 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 05:24:50.502346 5630 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 05:24:50.502385 5623 server_base.cc:1034] running on GCE node
W20260812 05:24:50.502385 5629 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 05:24:50.502791 5623 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 05:24:50.503057 5623 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 05:24:50.504201 5623 hybrid_clock.cc:648] HybridClock initialized: now 1786512290504185 us; error 32 us; skew 500 ppm
Aug 12 05:24:50 dist-test-slave-42z9 krb5kdc[5602](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512290, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.100.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 05:24:50.511621 5623 init.cc:377] Logged in from keytab as kudu/127.31.100.62@KRBTEST.COM (short username kudu)
I20260812 05:24:50.512024 5623 webserver.cc:460] Webserver started at http://127.31.100.62:44707/ using document root <none> and password file <none>
I20260812 05:24:50.512228 5623 fs_manager.cc:359] Metadata directory not provided
I20260812 05:24:50.512277 5623 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 05:24:50.512401 5623 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 05:24:50.513518 5623 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786512017613366-32144-0/minicluster-data/master-0/data/instance:
uuid: "2caa43b8b9c74725889e97fc4a8e26d1"
format_stamp: "Formatted at 2026-08-12 05:24:50 on dist-test-slave-42z9"
server_key: "c721f7091ea7eccb70c6f8c957e38390"
server_key_iv: "76d89f25224641c7f187d3580a5ac9e8"
server_key_version: "encryptionkey@0"
I20260812 05:24:50.513862 5623 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786512017613366-32144-0/minicluster-data/master-0/wal/instance:
uuid: "2caa43b8b9c74725889e97fc4a8e26d1"
format_stamp: "Formatted at 2026-08-12 05:24:50 on dist-test-slave-42z9"
server_key: "c721f7091ea7eccb70c6f8c957e38390"
server_key_iv: "76d89f25224641c7f187d3580a5ac9e8"
server_key_version: "encryptionkey@0"
I20260812 05:24:50.515085 5623 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260812 05:24:50.515727 5651 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 05:24:50.515878 5623 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260812 05:24:50.515966 5623 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786512017613366-32144-0/minicluster-data/master-0/data,/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786512017613366-32144-0/minicluster-data/master-0/wal
uuid: "2caa43b8b9c74725889e97fc4a8e26d1"
format_stamp: "Formatted at 2026-08-12 05:24:50 on dist-test-slave-42z9"
server_key: "c721f7091ea7eccb70c6f8c957e38390"
server_key_iv: "76d89f25224641c7f187d3580a5ac9e8"
server_key_version: "encryptionkey@0"
I20260812 05:24:50.516023 5623 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786512017613366-32144-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786512017613366-32144-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786512017613366-32144-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 05:24:50.529121 5623 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 05:24:50.531205 5623 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 05:24:50.531338 5623 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 05:24:50.535228 5623 rpc_server.cc:307] RPC server started. Bound to: 127.31.100.62:38737
I20260812 05:24:50.535259 5740 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.100.62:38737 every 8 connection(s)
I20260812 05:24:50.535574 5623 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786512017613366-32144-0/minicluster-data/master-0/data/info.pb
I20260812 05:24:50.536509 5742 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 05:24:50.538038 32144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0waFvT/build/release/bin/kudu as pid 5623
I20260812 05:24:50.538117 32144 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786512017613366-32144-0/minicluster-data/master-0/wal/instance
I20260812 05:24:50.538021 5742 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.100.62" port: 38737 } has no permanent_uuid. Determining permanent_uuid...
I20260812 05:24:50.538297 32144 external_mini_cluster.cc:1428] Setting key ed0bdd23348dc6e15aecd2e37dc9a9ba
Aug 12 05:24:50 dist-test-slave-42z9 krb5kdc[5602](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512290, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.100.62@KRBTEST.COM for kudu/127.31.100.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.31.100.61@KRBTEST.COM; defaulting to no policy
I20260812 05:24:50.548296 5749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:24:50.538729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46488 (local address 127.31.100.62:38737)
0812 05:24:50.538900 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0812 05:24:50.538904 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 05:24:50.538967 (+ 63us) server_negotiation.cc:407] Connection header received
0812 05:24:50.539639 (+ 672us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:24:50.539641 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:24:50.539660 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:24:50.539742 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:24:50.540417 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:24:50.541658 (+ 1241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:24:50.542821 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:24:50.543178 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:24:50.543255 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:24:50.545581 (+ 2326us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 05:24:50.545588 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 05:24:50.545589 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 05:24:50.545595 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0812 05:24:50.546973 (+ 1378us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:24:50.547314 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:24:50.547316 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:24:50.547316 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:24:50.547385 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:24:50.547548 (+ 163us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:24:50.547548 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:24:50.547549 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:24:50.547778 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 05:24:50.547836 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0812 05:24:50.548038 (+ 202us) server_negotiation.cc:299] Negotiation successful
0812 05:24:50.548125 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":57,"threads_started":1}
I20260812 05:24:50.548302 5744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:24:50.538673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.62:38737 (local address 127.0.0.1:46488)
0812 05:24:50.538841 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0812 05:24:50.538886 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0812 05:24:50.539478 (+ 592us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:24:50.539767 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:24:50.539769 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:24:50.540093 (+ 324us) client_negotiation.cc:189] Negotiated authn=SASL
0812 05:24:50.540317 (+ 224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:24:50.540322 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:24:50.541742 (+ 1420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:24:50.541743 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:24:50.542746 (+ 1003us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:24:50.542750 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:24:50.543278 (+ 528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:24:50.543280 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:24:50.543360 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:24:50.543859 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 05:24:50.543870 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0812 05:24:50.545504 (+ 1634us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 05:24:50.547045 (+ 1541us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 05:24:50.547053 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 05:24:50.547055 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 05:24:50.547252 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 05:24:50.547431 (+ 179us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 05:24:50.547432 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 05:24:50.547432 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 05:24:50.547502 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 05:24:50.547844 (+ 342us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 05:24:50.547852 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 05:24:50.547924 (+ 72us) client_negotiation.cc:769] Sending connection context
0812 05:24:50.548003 (+ 79us) client_negotiation.cc:240] Negotiation successful
0812 05:24:50.548190 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"mutex_wait_us":73,"thread_start_us":90,"threads_started":1}
I20260812 05:24:50.549095 5742 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.100.61" port: 42987 } has no permanent_uuid. Determining permanent_uuid...
I20260812 05:24:50.549396 5744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:24:50.549288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.61:42987 (local address 127.0.0.1:50818)
0812 05:24:50.549334 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0812 05:24:50.549377 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.100.61:42987: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.100.61@KRBTEST.COM" created.
W20260812 05:24:50.549530 5653 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.100.61:42987: connect: Connection refused (error 111)
I20260812 05:24:50.549918 5744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:24:50.549827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.61:42987 (local address 127.0.0.1:50828)
0812 05:24:50.549866 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0812 05:24:50.549896 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.100.61:42987: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260812 05:24:50.550091 5742 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.100.61:42987: Network error: Client connection negotiation failed: client connection to 127.31.100.61:42987: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.100.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.61.keytab.
Entry for principal kudu/127.31.100.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.61.keytab.
I20260812 05:24:50.561930 32144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0waFvT/build/release/bin/kudu
/tmp/dist-test-task0waFvT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786512017613366-32144-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786512017613366-32144-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786512017613366-32144-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786512017613366-32144-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.100.61:42987
--webserver_interface=127.31.100.61
--webserver_port=0
--keytab_file=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.61.keytab
--principal=kudu/127.31.100.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.100.20:40157
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.100.62:38737,127.31.100.61:42987,127.31.100.60:44447
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786512017613366-32144-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786512017613366-32144-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786512017613366-32144-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 05:24:50.599295 5742 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.100.61" port: 42987 } attempt: 1
I20260812 05:24:50.599795 5744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:24:50.599636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.61:42987 (local address 127.0.0.1:50836)
0812 05:24:50.599703 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0812 05:24:50.599754 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.100.61:42987: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260812 05:24:50.600123 5744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:24:50.600035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.61:42987 (local address 127.0.0.1:50838)
0812 05:24:50.600071 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0812 05:24:50.600096 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.100.61:42987: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
W20260812 05:24:50.600214 5742 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.100.61:42987: Network error: Client connection negotiation failed: client connection to 127.31.100.61:42987: connect: Connection refused (error 111)
W20260812 05:24:50.641623 5756 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260812 05:24:50.641809 5756 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 05:24:50.641825 5756 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 05:24:50.641846 5756 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 05:24:50.642952 5756 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 05:24:50.642992 5756 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 05:24:50.643004 5756 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 05:24:50.643028 5756 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 05:24:50.643050 5756 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 05:24:50.644513 5756 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.100.20:40157
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786512017613366-32144-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786512017613366-32144-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.31.100.62:38737,127.31.100.61:42987,127.31.100.60:44447
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.61.keytab
--principal=kudu/127.31.100.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.100.61:42987
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786512017613366-32144-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.31.100.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5756
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786512017613366-32144-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 04:20:34 UTC on ccycloud.quasar-phqdqp.root.comops.site
build id 4581871
I20260812 05:24:50.644742 5756 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 05:24:50.644948 5756 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 05:24:50.649539 5766 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 05:24:50.649583 5756 server_base.cc:1034] running on GCE node
W20260812 05:24:50.649541 5767 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 05:24:50.649546 5769 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 05:24:50.649974 5756 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 05:24:50.650262 5756 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 05:24:50.651403 5756 hybrid_clock.cc:648] HybridClock initialized: now 1786512290651388 us; error 28 us; skew 500 ppm
Aug 12 05:24:50 dist-test-slave-42z9 krb5kdc[5602](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512290, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.100.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 05:24:50.653697 5756 init.cc:377] Logged in from keytab as kudu/127.31.100.61@KRBTEST.COM (short username kudu)
I20260812 05:24:50.654098 5756 webserver.cc:460] Webserver started at http://127.31.100.61:45759/ using document root <none> and password file <none>
I20260812 05:24:50.654304 5756 fs_manager.cc:359] Metadata directory not provided
I20260812 05:24:50.654351 5756 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 05:24:50.654474 5756 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 05:24:50.655433 5756 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786512017613366-32144-0/minicluster-data/master-1/data/instance:
uuid: "fe28fc3af9c24c29b272bb8c54f875ea"
format_stamp: "Formatted at 2026-08-12 05:24:50 on dist-test-slave-42z9"
server_key: "c2206622eec455086d75498af6833d15"
server_key_iv: "f514d9e05d97683e912d1203bf20c8fd"
server_key_version: "encryptionkey@0"
I20260812 05:24:50.655753 5756 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786512017613366-32144-0/minicluster-data/master-1/wal/instance:
uuid: "fe28fc3af9c24c29b272bb8c54f875ea"
format_stamp: "Formatted at 2026-08-12 05:24:50 on dist-test-slave-42z9"
server_key: "c2206622eec455086d75498af6833d15"
server_key_iv: "f514d9e05d97683e912d1203bf20c8fd"
server_key_version: "encryptionkey@0"
I20260812 05:24:50.656926 5756 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260812 05:24:50.657574 5782 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 05:24:50.657709 5756 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260812 05:24:50.657793 5756 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786512017613366-32144-0/minicluster-data/master-1/data,/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786512017613366-32144-0/minicluster-data/master-1/wal
uuid: "fe28fc3af9c24c29b272bb8c54f875ea"
format_stamp: "Formatted at 2026-08-12 05:24:50 on dist-test-slave-42z9"
server_key: "c2206622eec455086d75498af6833d15"
server_key_iv: "f514d9e05d97683e912d1203bf20c8fd"
server_key_version: "encryptionkey@0"
I20260812 05:24:50.657855 5756 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786512017613366-32144-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786512017613366-32144-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786512017613366-32144-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 05:24:50.668354 5742 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.100.61" port: 42987 } attempt: 2
I20260812 05:24:50.668709 5744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:24:50.668593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.61:42987 (local address 127.0.0.1:50854)
0812 05:24:50.668643 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0812 05:24:50.668679 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.100.61:42987: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260812 05:24:50.668969 5744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:24:50.668902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.61:42987 (local address 127.0.0.1:50862)
0812 05:24:50.668932 (+ 30us) negotiation.cc:107] Waiting for socket to connect
0812 05:24:50.668955 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.100.61:42987: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":20}
W20260812 05:24:50.669039 5742 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.100.61:42987: Network error: Client connection negotiation failed: client connection to 127.31.100.61:42987: connect: Connection refused (error 111)
I20260812 05:24:50.676759 5756 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 05:24:50.678764 5756 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 05:24:50.678871 5756 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 05:24:50.682542 5756 rpc_server.cc:307] RPC server started. Bound to: 127.31.100.61:42987
I20260812 05:24:50.682585 5864 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.100.61:42987 every 8 connection(s)
I20260812 05:24:50.682886 5756 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786512017613366-32144-0/minicluster-data/master-1/data/info.pb
I20260812 05:24:50.683836 5866 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 05:24:50.685281 5866 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.100.62" port: 38737 } has no permanent_uuid. Determining permanent_uuid...
I20260812 05:24:50.686206 32144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0waFvT/build/release/bin/kudu as pid 5756
I20260812 05:24:50.686296 32144 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786512017613366-32144-0/minicluster-data/master-1/wal/instance
I20260812 05:24:50.686455 32144 external_mini_cluster.cc:1428] Setting key e80a4c08c4ee7f22475f63a0dca9173f
Aug 12 05:24:50 dist-test-slave-42z9 krb5kdc[5602](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512290, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.100.61@KRBTEST.COM for kudu/127.31.100.62@KRBTEST.COM
I20260812 05:24:50.693430 5749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:24:50.685944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46500 (local address 127.31.100.62:38737)
0812 05:24:50.685994 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0812 05:24:50.685996 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 05:24:50.686187 (+ 191us) server_negotiation.cc:407] Connection header received
0812 05:24:50.686723 (+ 536us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:24:50.686724 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:24:50.686736 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:24:50.686787 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:24:50.687360 (+ 573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:24:50.687827 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:24:50.688753 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:24:50.688950 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:24:50.689007 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:24:50.690987 (+ 1980us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 05:24:50.690989 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 05:24:50.690990 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 05:24:50.690998 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0812 05:24:50.692208 (+ 1210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:24:50.692568 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:24:50.692570 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:24:50.692570 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:24:50.692625 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:24:50.692837 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:24:50.692838 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:24:50.692838 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:24:50.692947 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 05:24:50.693003 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0812 05:24:50.693257 (+ 254us) server_negotiation.cc:299] Negotiation successful
0812 05:24:50.693341 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260812 05:24:50.693608 5873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:24:50.685909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.62:38737 (local address 127.0.0.1:46500)
0812 05:24:50.686076 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0812 05:24:50.686111 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0812 05:24:50.686629 (+ 518us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:24:50.686852 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:24:50.686856 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:24:50.687085 (+ 229us) client_negotiation.cc:189] Negotiated authn=SASL
0812 05:24:50.687271 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:24:50.687275 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:24:50.687908 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:24:50.687909 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:24:50.688671 (+ 762us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:24:50.688673 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:24:50.689042 (+ 369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:24:50.689043 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:24:50.689161 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:24:50.689584 (+ 423us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 05:24:50.689592 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0812 05:24:50.690897 (+ 1305us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 05:24:50.692285 (+ 1388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 05:24:50.692292 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 05:24:50.692294 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 05:24:50.692501 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 05:24:50.692680 (+ 179us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 05:24:50.692681 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 05:24:50.692681 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 05:24:50.692781 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 05:24:50.693002 (+ 221us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 05:24:50.693008 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 05:24:50.693141 (+ 133us) client_negotiation.cc:769] Sending connection context
0812 05:24:50.693227 (+ 86us) client_negotiation.cc:240] Negotiation successful
0812 05:24:50.693369 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":94,"threads_started":1}
WARNING: no policy specified for kudu/127.31.100.60@KRBTEST.COM; defaulting to no policy
I20260812 05:24:50.694111 5866 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.100.61" port: 42987 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.100.60@KRBTEST.COM" created.
Aug 12 05:24:50 dist-test-slave-42z9 krb5kdc[5602](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512290, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.100.61@KRBTEST.COM for kudu/127.31.100.61@KRBTEST.COM
I20260812 05:24:50.700934 5873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:24:50.694331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.61:42987 (local address 127.0.0.1:50864)
0812 05:24:50.694372 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0812 05:24:50.694382 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0812 05:24:50.694421 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:24:50.694636 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:24:50.694636 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:24:50.694769 (+ 133us) client_negotiation.cc:189] Negotiated authn=SASL
0812 05:24:50.694835 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:24:50.694837 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:24:50.695610 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:24:50.695610 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:24:50.696113 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:24:50.696115 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:24:50.696700 (+ 585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:24:50.696701 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:24:50.696779 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:24:50.697089 (+ 310us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 05:24:50.697094 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0812 05:24:50.698388 (+ 1294us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 05:24:50.699913 (+ 1525us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 05:24:50.699914 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 05:24:50.699914 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 05:24:50.700127 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 05:24:50.700305 (+ 178us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 05:24:50.700305 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 05:24:50.700306 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 05:24:50.700369 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 05:24:50.700602 (+ 233us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 05:24:50.700603 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 05:24:50.700659 (+ 56us) client_negotiation.cc:769] Sending connection context
0812 05:24:50.700729 (+ 70us) client_negotiation.cc:240] Negotiation successful
0812 05:24:50.700825 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26}
I20260812 05:24:50.700989 5875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:24:50.694367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50864 (local address 127.31.100.61:42987)
0812 05:24:50.694492 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0812 05:24:50.694514 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0812 05:24:50.694529 (+ 15us) server_negotiation.cc:407] Connection header received
0812 05:24:50.694549 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:24:50.694551 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:24:50.694561 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:24:50.694626 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:24:50.694897 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:24:50.695551 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:24:50.696173 (+ 622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:24:50.696619 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:24:50.696690 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:24:50.698465 (+ 1775us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 05:24:50.698473 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 05:24:50.698473 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 05:24:50.698479 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0812 05:24:50.699827 (+ 1348us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:24:50.700198 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:24:50.700200 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:24:50.700200 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:24:50.700250 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:24:50.700420 (+ 170us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:24:50.700421 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:24:50.700421 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:24:50.700539 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 05:24:50.700596 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0812 05:24:50.700769 (+ 173us) server_negotiation.cc:299] Negotiation successful
0812 05:24:50.700872 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":41,"server-negotiator.queue_time_us":102,"thread_start_us":75,"threads_started":1}
I20260812 05:24:50.701690 5866 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.100.60" port: 44447 } has no permanent_uuid. Determining permanent_uuid...
I20260812 05:24:50.702018 5873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:24:50.701886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.60:44447 (local address 127.0.0.1:49768)
0812 05:24:50.701930 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0812 05:24:50.701984 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.100.60:44447: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260812 05:24:50.702186 5784 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.100.60:44447: connect: Connection refused (error 111)
I20260812 05:24:50.702569 5873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:24:50.702470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.60:44447 (local address 127.0.0.1:49776)
0812 05:24:50.702515 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0812 05:24:50.702546 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.100.60:44447: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260812 05:24:50.702669 5866 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.100.60:44447: Network error: Client connection negotiation failed: client connection to 127.31.100.60:44447: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.100.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.60.keytab.
Entry for principal kudu/127.31.100.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.60.keytab.
I20260812 05:24:50.709664 32144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0waFvT/build/release/bin/kudu
/tmp/dist-test-task0waFvT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786512017613366-32144-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786512017613366-32144-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786512017613366-32144-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786512017613366-32144-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.100.60:44447
--webserver_interface=127.31.100.60
--webserver_port=0
--keytab_file=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.60.keytab
--principal=kudu/127.31.100.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.100.20:40157
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.100.62:38737,127.31.100.61:42987,127.31.100.60:44447
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786512017613366-32144-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786512017613366-32144-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786512017613366-32144-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 05:24:50.751888 5866 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.100.60" port: 44447 } attempt: 1
I20260812 05:24:50.752367 5873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:24:50.752232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.60:44447 (local address 127.0.0.1:49792)
0812 05:24:50.752291 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0812 05:24:50.752334 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.100.60:44447: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260812 05:24:50.752676 5873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:24:50.752593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.60:44447 (local address 127.0.0.1:49804)
0812 05:24:50.752632 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0812 05:24:50.752656 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.100.60:44447: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260812 05:24:50.752789 5866 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.100.60:44447: Network error: Client connection negotiation failed: client connection to 127.31.100.60:44447: connect: Connection refused (error 111)
I20260812 05:24:50.760176 5742 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.100.61" port: 42987 } attempt: 3
Aug 12 05:24:50 dist-test-slave-42z9 krb5kdc[5602](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512290, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.100.62@KRBTEST.COM for kudu/127.31.100.61@KRBTEST.COM
I20260812 05:24:50.767246 5744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:24:50.760432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.61:42987 (local address 127.0.0.1:50880)
0812 05:24:50.760487 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0812 05:24:50.760497 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0812 05:24:50.760548 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:24:50.760673 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:24:50.760674 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:24:50.760832 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0812 05:24:50.760902 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:24:50.760904 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:24:50.761916 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:24:50.761917 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:24:50.762443 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:24:50.762445 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:24:50.762949 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:24:50.762950 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:24:50.763013 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:24:50.763438 (+ 425us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 05:24:50.763443 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0812 05:24:50.764766 (+ 1323us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 05:24:50.766398 (+ 1632us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 05:24:50.766399 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 05:24:50.766399 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 05:24:50.766565 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 05:24:50.766747 (+ 182us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 05:24:50.766747 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 05:24:50.766748 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 05:24:50.766794 (+ 46us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 05:24:50.767026 (+ 232us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 05:24:50.767027 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 05:24:50.767065 (+ 38us) client_negotiation.cc:769] Sending connection context
0812 05:24:50.767108 (+ 43us) client_negotiation.cc:240] Negotiation successful
0812 05:24:50.767180 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260812 05:24:50.767297 5875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:24:50.760490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50880 (local address 127.31.100.61:42987)
0812 05:24:50.760545 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0812 05:24:50.760547 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 05:24:50.760557 (+ 10us) server_negotiation.cc:407] Connection header received
0812 05:24:50.760605 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:24:50.760606 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:24:50.760617 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:24:50.760662 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:24:50.760990 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:24:50.761831 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:24:50.762517 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:24:50.762869 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:24:50.762941 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:24:50.764848 (+ 1907us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 05:24:50.764849 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 05:24:50.764850 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 05:24:50.764857 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 05:24:50.766321 (+ 1464us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:24:50.766628 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:24:50.766629 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:24:50.766629 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:24:50.766693 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:24:50.766842 (+ 149us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:24:50.766842 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:24:50.766842 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:24:50.766967 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 05:24:50.767038 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0812 05:24:50.767138 (+ 100us) server_negotiation.cc:299] Negotiation successful
0812 05:24:50.767214 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260812 05:24:50.767658 5742 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.100.60" port: 44447 } has no permanent_uuid. Determining permanent_uuid...
I20260812 05:24:50.767936 5744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:24:50.767858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.60:44447 (local address 127.0.0.1:49810)
0812 05:24:50.767889 (+ 31us) negotiation.cc:107] Waiting for socket to connect
0812 05:24:50.767915 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.100.60:44447: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":16}
I20260812 05:24:50.768198 5744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:24:50.768129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.60:44447 (local address 127.0.0.1:49816)
0812 05:24:50.768161 (+ 32us) negotiation.cc:107] Waiting for socket to connect
0812 05:24:50.768183 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.100.60:44447: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
W20260812 05:24:50.768277 5742 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.100.60:44447: Network error: Client connection negotiation failed: client connection to 127.31.100.60:44447: connect: Connection refused (error 111)
W20260812 05:24:50.790544 5879 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260812 05:24:50.790740 5879 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 05:24:50.790761 5879 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 05:24:50.790788 5879 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 05:24:50.791884 5879 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 05:24:50.791924 5879 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 05:24:50.791944 5879 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 05:24:50.791958 5879 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 05:24:50.791975 5879 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 05:24:50.793462 5879 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.100.20:40157
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786512017613366-32144-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786512017613366-32144-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.31.100.62:38737,127.31.100.61:42987,127.31.100.60:44447
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786512017613366-32144-0/krb5kdc/kudu_127.31.100.60.keytab
--principal=kudu/127.31.100.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.100.60:44447
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786512017613366-32144-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.31.100.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5879
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786512017613366-32144-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 04:20:34 UTC on ccycloud.quasar-phqdqp.root.comops.site
build id 4581871
I20260812 05:24:50.793708 5879 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 05:24:50.793905 5879 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 05:24:50.798354 5891 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 05:24:50.798393 5879 server_base.cc:1034] running on GCE node
W20260812 05:24:50.798473 5893 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 05:24:50.798367 5890 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 05:24:50.798779 5879 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 05:24:50.799018 5879 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 05:24:50.799407 5742 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.100.60" port: 44447 } attempt: 1
I20260812 05:24:50.799743 5744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:24:50.799638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.60:44447 (local address 127.0.0.1:49832)
0812 05:24:50.799687 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0812 05:24:50.799718 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.100.60:44447: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260812 05:24:50.799962 5744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:24:50.799911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.60:44447 (local address 127.0.0.1:49844)
0812 05:24:50.799931 (+ 20us) negotiation.cc:107] Waiting for socket to connect
0812 05:24:50.799950 (+ 19us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.100.60:44447: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":11}
W20260812 05:24:50.800030 5742 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.100.60:44447: Network error: Client connection negotiation failed: client connection to 127.31.100.60:44447: connect: Connection refused (error 111)
I20260812 05:24:50.800148 5879 hybrid_clock.cc:648] HybridClock initialized: now 1786512290800134 us; error 27 us; skew 500 ppm
Aug 12 05:24:50 dist-test-slave-42z9 krb5kdc[5602](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512290, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.100.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 05:24:50.802438 5879 init.cc:377] Logged in from keytab as kudu/127.31.100.60@KRBTEST.COM (short username kudu)
I20260812 05:24:50.802847 5879 webserver.cc:460] Webserver started at http://127.31.100.60:37999/ using document root <none> and password file <none>
I20260812 05:24:50.803058 5879 fs_manager.cc:359] Metadata directory not provided
I20260812 05:24:50.803102 5879 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 05:24:50.803220 5879 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 05:24:50.804176 5879 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786512017613366-32144-0/minicluster-data/master-2/data/instance:
uuid: "6e028db49a0243dd9baf3c86319ab253"
format_stamp: "Formatted at 2026-08-12 05:24:50 on dist-test-slave-42z9"
server_key: "be188c81512c841bf80f0e59232d8d26"
server_key_iv: "646b23f5aeeab4c57d2372e3e217d781"
server_key_version: "encryptionkey@0"
I20260812 05:24:50.804486 5879 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786512017613366-32144-0/minicluster-data/master-2/wal/instance:
uuid: "6e028db49a0243dd9baf3c86319ab253"
format_stamp: "Formatted at 2026-08-12 05:24:50 on dist-test-slave-42z9"
server_key: "be188c81512c841bf80f0e59232d8d26"
server_key_iv: "646b23f5aeeab4c57d2372e3e217d781"
server_key_version: "encryptionkey@0"
I20260812 05:24:50.805770 5879 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260812 05:24:50.806401 5905 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 05:24:50.806550 5879 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260812 05:24:50.806634 5879 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786512017613366-32144-0/minicluster-data/master-2/data,/tmp/dist-test-task0waFvT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786512017613366-32144-0/minicluster-data/master-2/wal
uuid: "6e028db49a0243dd9baf3c86319ab253"
format_stamp: "Formatted at 2026-08-12 05:24:50 on dist-test-slave-42z9"
server_key: "be188c81512c841bf80f0e59232d8d26"
server_key_iv: "646b23f5aeeab4c57d2372e3e217d781"
server_key_version: "encryptionkey@0"
I20260812 05:24:50.806691 5879 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786512017613366-32144-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786512017613366-32144-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786512017613366-32144-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 05:24:50.817795 5879 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 05:24:50.819453 5879 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 05:24:50.819584 5879 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 05:24:50.820938 5866 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.100.60" port: 44447 } attempt: 2
I20260812 05:24:50.821312 5873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:24:50.821166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.60:44447 (local address 127.0.0.1:49860)
0812 05:24:50.821234 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0812 05:24:50.821276 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.100.60:44447: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260812 05:24:50.821696 5873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:24:50.821584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.60:44447 (local address 127.0.0.1:49872)
0812 05:24:50.821640 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0812 05:24:50.821672 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.100.60:44447: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260812 05:24:50.821799 5866 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.100.60:44447: Network error: Client connection negotiation failed: client connection to 127.31.100.60:44447: connect: Connection refused (error 111)
I20260812 05:24:50.823329 5879 rpc_server.cc:307] RPC server started. Bound to: 127.31.100.60:44447
I20260812 05:24:50.823357 5985 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.100.60:44447 every 8 connection(s)
I20260812 05:24:50.823648 5879 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786512017613366-32144-0/minicluster-data/master-2/data/info.pb
I20260812 05:24:50.824538 5986 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 05:24:50.825982 5986 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.100.62" port: 38737 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 05:24:50 dist-test-slave-42z9 krb5kdc[5602](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512290, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.100.60@KRBTEST.COM for kudu/127.31.100.62@KRBTEST.COM
I20260812 05:24:50.833199 5749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:24:50.826571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46514 (local address 127.31.100.62:38737)
0812 05:24:50.826619 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0812 05:24:50.826621 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 05:24:50.826806 (+ 185us) server_negotiation.cc:407] Connection header received
0812 05:24:50.827245 (+ 439us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:24:50.827246 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:24:50.827256 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:24:50.827290 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:24:50.827796 (+ 506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:24:50.828247 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:24:50.829090 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:24:50.829262 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:24:50.829304 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:24:50.831173 (+ 1869us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 05:24:50.831175 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 05:24:50.831175 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 05:24:50.831182 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 05:24:50.832216 (+ 1034us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:24:50.832544 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:24:50.832544 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:24:50.832545 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:24:50.832588 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:24:50.832757 (+ 169us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:24:50.832758 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:24:50.832758 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:24:50.832848 (+ 90us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 05:24:50.832889 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0812 05:24:50.833085 (+ 196us) server_negotiation.cc:299] Negotiation successful
0812 05:24:50.833141 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260812 05:24:50.833364 5989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:24:50.826559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.62:38737 (local address 127.0.0.1:46514)
0812 05:24:50.826702 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0812 05:24:50.826733 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0812 05:24:50.827162 (+ 429us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:24:50.827343 (+ 181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:24:50.827346 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:24:50.827558 (+ 212us) client_negotiation.cc:189] Negotiated authn=SASL
0812 05:24:50.827738 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:24:50.827742 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:24:50.828304 (+ 562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:24:50.828304 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:24:50.829031 (+ 727us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:24:50.829033 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:24:50.829326 (+ 293us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:24:50.829326 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:24:50.829413 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:24:50.829854 (+ 441us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 05:24:50.829862 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0812 05:24:50.831084 (+ 1222us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 05:24:50.832281 (+ 1197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 05:24:50.832286 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 05:24:50.832288 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 05:24:50.832489 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 05:24:50.832631 (+ 142us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 05:24:50.832631 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 05:24:50.832632 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 05:24:50.832717 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 05:24:50.832888 (+ 171us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 05:24:50.832891 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 05:24:50.832991 (+ 100us) client_negotiation.cc:769] Sending connection context
0812 05:24:50.833062 (+ 71us) client_negotiation.cc:240] Negotiation successful
0812 05:24:50.833177 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":79,"threads_started":1}
I20260812 05:24:50.833802 32144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0waFvT/build/release/bin/kudu as pid 5879
I20260812 05:24:50.833899 32144 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0waFvT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786512017613366-32144-0/minicluster-data/master-2/wal/instance
I20260812 05:24:50.833909 5986 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.100.61" port: 42987 } has no permanent_uuid. Determining permanent_uuid...
I20260812 05:24:50.834103 32144 external_mini_cluster.cc:1428] Setting key 9432a6ab7b06ae31d22524730907a70c
Aug 12 05:24:50 dist-test-slave-42z9 krb5kdc[5602](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512290, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.100.60@KRBTEST.COM for kudu/127.31.100.61@KRBTEST.COM
Aug 12 05:24:50 dist-test-slave-42z9 krb5kdc[5602](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512290, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.100.60@KRBTEST.COM
I20260812 05:24:50.840288 5989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:24:50.834106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.61:42987 (local address 127.0.0.1:50884)
0812 05:24:50.834145 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0812 05:24:50.834156 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0812 05:24:50.834191 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:24:50.834335 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:24:50.834336 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:24:50.834473 (+ 137us) client_negotiation.cc:189] Negotiated authn=SASL
0812 05:24:50.834534 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:24:50.834536 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:24:50.835377 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:24:50.835377 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:24:50.836102 (+ 725us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:24:50.836104 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:24:50.836426 (+ 322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:24:50.836427 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:24:50.836493 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:24:50.836811 (+ 318us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 05:24:50.836815 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0812 05:24:50.838048 (+ 1233us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 05:24:50.839387 (+ 1339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 05:24:50.839388 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 05:24:50.839388 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 05:24:50.839554 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 05:24:50.839756 (+ 202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 05:24:50.839756 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 05:24:50.839756 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 05:24:50.839814 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 05:24:50.840025 (+ 211us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 05:24:50.840026 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 05:24:50.840069 (+ 43us) client_negotiation.cc:769] Sending connection context
0812 05:24:50.840126 (+ 57us) client_negotiation.cc:240] Negotiation successful
0812 05:24:50.840203 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25}
I20260812 05:24:50.840324 5875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:24:50.834179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50884 (local address 127.31.100.61:42987)
0812 05:24:50.834224 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0812 05:24:50.834226 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 05:24:50.834234 (+ 8us) server_negotiation.cc:407] Connection header received
0812 05:24:50.834259 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:24:50.834260 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:24:50.834270 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:24:50.834321 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:24:50.834600 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:24:50.835298 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:24:50.836176 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:24:50.836370 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:24:50.836417 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:24:50.838131 (+ 1714us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 05:24:50.838133 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 05:24:50.838133 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 05:24:50.838140 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 05:24:50.839298 (+ 1158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:24:50.839630 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:24:50.839631 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:24:50.839631 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:24:50.839689 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:24:50.839865 (+ 176us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:24:50.839865 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:24:50.839866 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:24:50.839964 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 05:24:50.840013 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0812 05:24:50.840167 (+ 154us) server_negotiation.cc:299] Negotiation successful
0812 05:24:50.840245 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260812 05:24:50.840853 5986 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.100.60" port: 44447 } has no permanent_uuid. Determining permanent_uuid...
I20260812 05:24:50.842444 5990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:24:50.834420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49884 (local address 127.31.100.60:44447)
0812 05:24:50.834552 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0812 05:24:50.834556 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 05:24:50.834732 (+ 176us) server_negotiation.cc:407] Connection header received
0812 05:24:50.834756 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:24:50.834757 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:24:50.834768 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:24:50.834823 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:24:50.835481 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:24:50.836412 (+ 931us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:24:50.837099 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:24:50.837476 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:24:50.837543 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:24:50.839814 (+ 2271us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 05:24:50.839822 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 05:24:50.839822 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 05:24:50.839828 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0812 05:24:50.841422 (+ 1594us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:24:50.841747 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:24:50.841750 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:24:50.841751 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:24:50.841811 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:24:50.841960 (+ 149us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:24:50.841961 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:24:50.841961 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:24:50.842113 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 05:24:50.842246 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0812 05:24:50.842283 (+ 37us) server_negotiation.cc:299] Negotiation successful
0812 05:24:50.842354 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103,"thread_start_us":69,"threads_started":1}
Aug 12 05:24:50 dist-test-slave-42z9 krb5kdc[5602](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512290, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.100.60@KRBTEST.COM for kudu/127.31.100.60@KRBTEST.COM
Aug 12 05:24:50 dist-test-slave-42z9 krb5kdc[5602](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512290, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.100.62@KRBTEST.COM
I20260812 05:24:50.848076 5989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:24:50.841099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.60:44447 (local address 127.0.0.1:49888)
0812 05:24:50.841252 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0812 05:24:50.841262 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0812 05:24:50.841300 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:24:50.841416 (+ 116us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:24:50.841417 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:24:50.841562 (+ 145us) client_negotiation.cc:189] Negotiated authn=SASL
0812 05:24:50.841585 (+ 23us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:24:50.841587 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:24:50.842365 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:24:50.842366 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:24:50.842970 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:24:50.842972 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:24:50.843449 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:24:50.843449 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:24:50.843500 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:24:50.843850 (+ 350us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 05:24:50.843854 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0812 05:24:50.845113 (+ 1259us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 05:24:50.847108 (+ 1995us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 05:24:50.847109 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 05:24:50.847109 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 05:24:50.847301 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 05:24:50.847505 (+ 204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 05:24:50.847506 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 05:24:50.847506 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 05:24:50.847569 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 05:24:50.847796 (+ 227us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 05:24:50.847797 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 05:24:50.847832 (+ 35us) client_negotiation.cc:769] Sending connection context
0812 05:24:50.847881 (+ 49us) client_negotiation.cc:240] Negotiation successful
0812 05:24:50.847964 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141}
I20260812 05:24:50.848171 5993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:24:50.841158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49888 (local address 127.31.100.60:44447)
0812 05:24:50.841276 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0812 05:24:50.841278 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 05:24:50.841287 (+ 9us) server_negotiation.cc:407] Connection header received
0812 05:24:50.841340 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:24:50.841341 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:24:50.841349 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:24:50.841414 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:24:50.841646 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:24:50.842292 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:24:50.843038 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:24:50.843394 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:24:50.843443 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:24:50.845185 (+ 1742us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 05:24:50.845186 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 05:24:50.845186 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 05:24:50.845192 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0812 05:24:50.847032 (+ 1840us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:24:50.847390 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:24:50.847391 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:24:50.847391 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:24:50.847445 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:24:50.847638 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:24:50.847639 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:24:50.847639 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:24:50.847736 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 05:24:50.847797 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0812 05:24:50.847919 (+ 122us) server_negotiation.cc:299] Negotiation successful
0812 05:24:50.848015 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":23,"server-negotiator.queue_time_us":103,"thread_start_us":68,"threads_started":1}
I20260812 05:24:50.849407 5986 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6e028db49a0243dd9baf3c86319ab253: Bootstrap starting.
I20260812 05:24:50.850188 5986 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6e028db49a0243dd9baf3c86319ab253: Neither blocks nor log segments found. Creating new log.
I20260812 05:24:50.850351 5749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:24:50.843206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46530 (local address 127.31.100.62:38737)
0812 05:24:50.843243 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0812 05:24:50.843245 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 05:24:50.843252 (+ 7us) server_negotiation.cc:407] Connection header received
0812 05:24:50.843270 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:24:50.843271 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:24:50.843279 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:24:50.843323 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:24:50.843764 (+ 441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:24:50.844432 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:24:50.845314 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:24:50.845622 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:24:50.845682 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:24:50.847727 (+ 2045us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 05:24:50.847728 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 05:24:50.847728 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 05:24:50.847734 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0812 05:24:50.849333 (+ 1599us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:24:50.849698 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:24:50.849699 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:24:50.849699 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:24:50.849747 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:24:50.849927 (+ 180us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:24:50.849928 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:24:50.849928 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:24:50.850014 (+ 86us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 05:24:50.850110 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0812 05:24:50.850213 (+ 103us) server_negotiation.cc:299] Negotiation successful
0812 05:24:50.850278 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260812 05:24:50.850498 5986 log.cc:824] T 00000000000000000000000000000000 P 6e028db49a0243dd9baf3c86319ab253: Log is configured to *not* fsync() on all Append() calls
I20260812 05:24:50.851315 5986 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6e028db49a0243dd9baf3c86319ab253: No bootstrap required, opened a new log
I20260812 05:24:50.853104 5986 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6e028db49a0243dd9baf3c86319ab253 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2caa43b8b9c74725889e97fc4a8e26d1" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 38737 } } peers { permanent_uuid: "fe28fc3af9c24c29b272bb8c54f875ea" member_type: VOTER last_known_addr { host: "127.31.100.61" port: 42987 } } peers { permanent_uuid: "6e028db49a0243dd9baf3c86319ab253" member_type: VOTER last_known_addr { host: "127.31.100.60" port: 44447 } }
I20260812 05:24:50.853233 5986 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6e028db49a0243dd9baf3c86319ab253 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e028db49a0243dd9baf3c86319ab253, State: Initialized, Role: FOLLOWER
I20260812 05:24:50.853294 5986 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6e028db49a0243dd9baf3c86319ab253 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2caa43b8b9c74725889e97fc4a8e26d1" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 38737 } } peers { permanent_uuid: "fe28fc3af9c24c29b272bb8c54f875ea" member_type: VOTER last_known_addr { host: "127.31.100.61" port: 42987 } } peers { permanent_uuid: "6e028db49a0243dd9baf3c86319ab253" member_type: VOTER last_known_addr { host: "127.31.100.60" port: 44447 } }
I20260812 05:24:50.853646 5996 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6e028db49a0243dd9baf3c86319ab253 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2caa43b8b9c74725889e97fc4a8e26d1" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 38737 } } peers { permanent_uuid: "fe28fc3af9c24c29b272bb8c54f875ea" member_type: VOTER last_known_addr { host: "127.31.100.61" port: 42987 } } peers { permanent_uuid: "6e028db49a0243dd9baf3c86319ab253" member_type: VOTER last_known_addr { host: "127.31.100.60" port: 44447 } } }
I20260812 05:24:50.853698 5986 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6e028db49a0243dd9baf3c86319ab253 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 05:24:50.853756 5996 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6e028db49a0243dd9baf3c86319ab253 [sys.catalog]: This master's current role is: FOLLOWER
Aug 12 05:24:50 dist-test-slave-42z9 krb5kdc[5602](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512290, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.100.61@KRBTEST.COM
W20260812 05:24:50.855667 6013 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6e028db49a0243dd9baf3c86319ab253: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 05:24:50.855760 6013 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 05:24:50.858235 5875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:24:50.850943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50888 (local address 127.31.100.61:42987)
0812 05:24:50.850976 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0812 05:24:50.850978 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 05:24:50.851213 (+ 235us) server_negotiation.cc:407] Connection header received
0812 05:24:50.851271 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:24:50.851272 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:24:50.851279 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:24:50.851312 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:24:50.851805 (+ 493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:24:50.852226 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:24:50.853080 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:24:50.853278 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:24:50.853331 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:24:50.855560 (+ 2229us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 05:24:50.855562 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 05:24:50.855562 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 05:24:50.855570 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0812 05:24:50.857097 (+ 1527us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:24:50.857502 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:24:50.857503 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:24:50.857504 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:24:50.857562 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:24:50.857756 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:24:50.857757 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:24:50.857757 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:24:50.857889 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 05:24:50.858046 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0812 05:24:50.858084 (+ 38us) server_negotiation.cc:299] Negotiation successful
0812 05:24:50.858153 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260812 05:24:50.856124 5879 master_runner.cc:186] Error getting master registration for 127.31.100.62:38737: OK, instance_id { permanent_uuid: "2caa43b8b9c74725889e97fc4a8e26d1" instance_seqno: 1786512290534133 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260812 05:24:50.858484 5879 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260812 05:24:50.859568 32144 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 05:24:50.860615 5701 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "2caa43b8b9c74725889e97fc4a8e26d1"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46530
I20260812 05:24:50.875149 5742 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.100.60" port: 44447 } attempt: 2
Aug 12 05:24:50 dist-test-slave-42z9 krb5kdc[5602](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512290, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.100.62@KRBTEST.COM for kudu/127.31.100.60@KRBTEST.COM
I20260812 05:24:50.882256 5744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:24:50.875394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.60:44447 (local address 127.0.0.1:49894)
0812 05:24:50.875441 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0812 05:24:50.875450 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0812 05:24:50.875488 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:24:50.875663 (+ 175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:24:50.875664 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:24:50.875831 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0812 05:24:50.875877 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:24:50.875878 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:24:50.876761 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:24:50.876762 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:24:50.877574 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:24:50.877576 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:24:50.878047 (+ 471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:24:50.878048 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:24:50.878100 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:24:50.878490 (+ 390us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 05:24:50.878495 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0812 05:24:50.879584 (+ 1089us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 05:24:50.881208 (+ 1624us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 05:24:50.881209 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 05:24:50.881209 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 05:24:50.881416 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 05:24:50.881627 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 05:24:50.881627 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 05:24:50.881628 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 05:24:50.881687 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 05:24:50.881972 (+ 285us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 05:24:50.881973 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 05:24:50.882014 (+ 41us) client_negotiation.cc:769] Sending connection context
0812 05:24:50.882073 (+ 59us) client_negotiation.cc:240] Negotiation successful
0812 05:24:50.882158 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260812 05:24:50.882285 5993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:24:50.875496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49894 (local address 127.31.100.60:44447)
0812 05:24:50.875546 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0812 05:24:50.875548 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 05:24:50.875556 (+ 8us) server_negotiation.cc:407] Connection header received
0812 05:24:50.875577 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:24:50.875578 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:24:50.875589 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:24:50.875651 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:24:50.875938 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:24:50.876686 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:24:50.877666 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:24:50.877980 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:24:50.878057 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:24:50.879675 (+ 1618us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 05:24:50.879677 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 05:24:50.879677 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 05:24:50.879684 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 05:24:50.881120 (+ 1436us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:24:50.881509 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:24:50.881510 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:24:50.881511 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:24:50.881567 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:24:50.881740 (+ 173us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:24:50.881741 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:24:50.881741 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:24:50.881913 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 05:24:50.881978 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0812 05:24:50.882115 (+ 137us) server_negotiation.cc:299] Negotiation successful
0812 05:24:50.882196 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260812 05:24:50.883729 5742 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2caa43b8b9c74725889e97fc4a8e26d1: Bootstrap starting.
I20260812 05:24:50.884346 5742 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2caa43b8b9c74725889e97fc4a8e26d1: Neither blocks nor log segments found. Creating new log.
I20260812 05:24:50.884615 5742 log.cc:824] T 00000000000000000000000000000000 P 2caa43b8b9c74725889e97fc4a8e26d1: Log is configured to *not* fsync() on all Append() calls
I20260812 05:24:50.885303 5742 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2caa43b8b9c74725889e97fc4a8e26d1: No bootstrap required, opened a new log
I20260812 05:24:50.886307 5742 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2caa43b8b9c74725889e97fc4a8e26d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2caa43b8b9c74725889e97fc4a8e26d1" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 38737 } } peers { permanent_uuid: "fe28fc3af9c24c29b272bb8c54f875ea" member_type: VOTER last_known_addr { host: "127.31.100.61" port: 42987 } } peers { permanent_uuid: "6e028db49a0243dd9baf3c86319ab253" member_type: VOTER last_known_addr { host: "127.31.100.60" port: 44447 } }
I20260812 05:24:50.886415 5742 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2caa43b8b9c74725889e97fc4a8e26d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2caa43b8b9c74725889e97fc4a8e26d1, State: Initialized, Role: FOLLOWER
I20260812 05:24:50.886492 5742 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2caa43b8b9c74725889e97fc4a8e26d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2caa43b8b9c74725889e97fc4a8e26d1" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 38737 } } peers { permanent_uuid: "fe28fc3af9c24c29b272bb8c54f875ea" member_type: VOTER last_known_addr { host: "127.31.100.61" port: 42987 } } peers { permanent_uuid: "6e028db49a0243dd9baf3c86319ab253" member_type: VOTER last_known_addr { host: "127.31.100.60" port: 44447 } }
I20260812 05:24:50.886735 6017 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2caa43b8b9c74725889e97fc4a8e26d1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2caa43b8b9c74725889e97fc4a8e26d1" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 38737 } } peers { permanent_uuid: "fe28fc3af9c24c29b272bb8c54f875ea" member_type: VOTER last_known_addr { host: "127.31.100.61" port: 42987 } } peers { permanent_uuid: "6e028db49a0243dd9baf3c86319ab253" member_type: VOTER last_known_addr { host: "127.31.100.60" port: 44447 } } }
I20260812 05:24:50.886788 5742 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2caa43b8b9c74725889e97fc4a8e26d1 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 05:24:50.886827 6017 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2caa43b8b9c74725889e97fc4a8e26d1 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 05:24:50.886929 5701 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2caa43b8b9c74725889e97fc4a8e26d1 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260812 05:24:50.887012 5701 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2caa43b8b9c74725889e97fc4a8e26d1 [term 0 FOLLOWER]: Advancing to term 1
I20260812 05:24:50.887764 5701 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2caa43b8b9c74725889e97fc4a8e26d1 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2caa43b8b9c74725889e97fc4a8e26d1" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 38737 } } peers { permanent_uuid: "fe28fc3af9c24c29b272bb8c54f875ea" member_type: VOTER last_known_addr { host: "127.31.100.61" port: 42987 } } peers { permanent_uuid: "6e028db49a0243dd9baf3c86319ab253" member_type: VOTER last_known_addr { host: "127.31.100.60" port: 44447 } }
I20260812 05:24:50.887997 5701 leader_election.cc:288] T 00000000000000000000000000000000 P 2caa43b8b9c74725889e97fc4a8e26d1 [CANDIDATE]: Term 1 election: Requested vote from peers fe28fc3af9c24c29b272bb8c54f875ea (127.31.100.61:42987), 6e028db49a0243dd9baf3c86319ab253 (127.31.100.60:44447)
I20260812 05:24:50.888298 5947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2caa43b8b9c74725889e97fc4a8e26d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "6e028db49a0243dd9baf3c86319ab253"
I20260812 05:24:50.888430 5947 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6e028db49a0243dd9baf3c86319ab253 [term 0 FOLLOWER]: Advancing to term 1
I20260812 05:24:50.889325 5947 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 6e028db49a0243dd9baf3c86319ab253 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2caa43b8b9c74725889e97fc4a8e26d1 in term 1.
W20260812 05:24:50.889760 6037 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2caa43b8b9c74725889e97fc4a8e26d1: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 05:24:50.889828 6037 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 05:24:50.890024 5653 leader_election.cc:302] T 00000000000000000000000000000000 P 2caa43b8b9c74725889e97fc4a8e26d1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2caa43b8b9c74725889e97fc4a8e26d1, 6e028db49a0243dd9baf3c86319ab253; no voters:
I20260812 05:24:50.890146 6017 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2caa43b8b9c74725889e97fc4a8e26d1 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 05:24:50.890269 6017 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2caa43b8b9c74725889e97fc4a8e26d1 [term 1 LEADER]: Becoming Leader. State: Replica: 2caa43b8b9c74725889e97fc4a8e26d1, State: Running, Role: LEADER
I20260812 05:24:50.890344 6017 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2caa43b8b9c74725889e97fc4a8e26d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2caa43b8b9c74725889e97fc4a8e26d1" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 38737 } } peers { permanent_uuid: "fe28fc3af9c24c29b272bb8c54f875ea" member_type: VOTER last_known_addr { host: "127.31.100.61" port: 42987 } } peers { permanent_uuid: "6e028db49a0243dd9baf3c86319ab253" member_type: VOTER last_known_addr { host: "127.31.100.60" port: 44447 } }
I20260812 05:24:50.890749 6017 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2caa43b8b9c74725889e97fc4a8e26d1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2caa43b8b9c74725889e97fc4a8e26d1. Latest consensus state: current_term: 1 leader_uuid: "2caa43b8b9c74725889e97fc4a8e26d1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2caa43b8b9c74725889e97fc4a8e26d1" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 38737 } } peers { permanent_uuid: "fe28fc3af9c24c29b272bb8c54f875ea" member_type: VOTER last_known_addr { host: "127.31.100.61" port: 42987 } } peers { permanent_uuid: "6e028db49a0243dd9baf3c86319ab253" member_type: VOTER last_known_addr { host: "127.31.100.60" port: 44447 } } }
I20260812 05:24:50.890812 6017 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2caa43b8b9c74725889e97fc4a8e26d1 [sys.catalog]: This master's current role is: LEADER
I20260812 05:24:50.891028 6049 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 05:24:50.891203 6049 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 05:24:50.889955 5623 master_runner.cc:186] Error getting master registration for 127.31.100.61:42987: OK, instance_id { permanent_uuid: "fe28fc3af9c24c29b272bb8c54f875ea" instance_seqno: 1786512290681518 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260812 05:24:50.892130 5623 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260812 05:24:50.897516 5947 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 6e028db49a0243dd9baf3c86319ab253 [term 1 FOLLOWER]: Refusing update from remote peer 2caa43b8b9c74725889e97fc4a8e26d1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 05:24:50.897830 6017 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 2caa43b8b9c74725889e97fc4a8e26d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e028db49a0243dd9baf3c86319ab253" member_type: VOTER last_known_addr { host: "127.31.100.60" port: 44447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 05:24:50.899197 5996 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6e028db49a0243dd9baf3c86319ab253 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2caa43b8b9c74725889e97fc4a8e26d1. Latest consensus state: current_term: 1 leader_uuid: "2caa43b8b9c74725889e97fc4a8e26d1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2caa43b8b9c74725889e97fc4a8e26d1" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 38737 } } peers { permanent_uuid: "fe28fc3af9c24c29b272bb8c54f875ea" member_type: VOTER last_known_addr { host: "127.31.100.61" port: 42987 } } peers { permanent_uuid: "6e028db49a0243dd9baf3c86319ab253" member_type: VOTER last_known_addr { host: "127.31.100.60" port: 44447 } } }
I20260812 05:24:50.899294 5996 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6e028db49a0243dd9baf3c86319ab253 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 05:24:50.904662 5875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:24:50.892809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50902 (local address 127.31.100.61:42987)
0812 05:24:50.892850 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0812 05:24:50.892852 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 05:24:50.893217 (+ 365us) server_negotiation.cc:407] Connection header received
0812 05:24:50.893350 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:24:50.893351 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:24:50.893361 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:24:50.893390 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:24:50.894515 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:24:50.895247 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:24:50.897010 (+ 1763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:24:50.897822 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:24:50.897871 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:24:50.900776 (+ 2905us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 05:24:50.900777 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 05:24:50.900778 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 05:24:50.900785 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 05:24:50.902486 (+ 1701us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:24:50.903314 (+ 828us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:24:50.903316 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:24:50.903316 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:24:50.903637 (+ 321us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:24:50.903965 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:24:50.903966 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:24:50.903966 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:24:50.904102 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 05:24:50.904171 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0812 05:24:50.904487 (+ 316us) server_negotiation.cc:299] Negotiation successful
0812 05:24:50.904572 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260812 05:24:50.905550 5749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:24:50.892086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46532 (local address 127.31.100.62:38737)
0812 05:24:50.892121 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0812 05:24:50.892123 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 05:24:50.893147 (+ 1024us) server_negotiation.cc:407] Connection header received
0812 05:24:50.893239 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:24:50.893240 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:24:50.893249 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:24:50.893276 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:24:50.893967 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:24:50.894697 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:24:50.899247 (+ 4550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:24:50.899536 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:24:50.899579 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:24:50.902423 (+ 2844us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 05:24:50.902424 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 05:24:50.902425 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 05:24:50.902432 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 05:24:50.904202 (+ 1770us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:24:50.904585 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:24:50.904586 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:24:50.904587 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:24:50.904643 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:24:50.904922 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:24:50.904923 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:24:50.904923 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:24:50.905050 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 05:24:50.905141 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0812 05:24:50.905340 (+ 199us) server_negotiation.cc:299] Negotiation successful
0812 05:24:50.905415 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260812 05:24:50.907306 6046 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2caa43b8b9c74725889e97fc4a8e26d1 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2caa43b8b9c74725889e97fc4a8e26d1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2caa43b8b9c74725889e97fc4a8e26d1" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 38737 } } peers { permanent_uuid: "fe28fc3af9c24c29b272bb8c54f875ea" member_type: VOTER last_known_addr { host: "127.31.100.61" port: 42987 } } peers { permanent_uuid: "6e028db49a0243dd9baf3c86319ab253" member_type: VOTER last_known_addr { host: "127.31.100.60" port: 44447 } } }
I20260812 05:24:50.907407 6046 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2caa43b8b9c74725889e97fc4a8e26d1 [sys.catalog]: This master's current role is: LEADER
I20260812 05:24:50.907403 5996 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6e028db49a0243dd9baf3c86319ab253 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2caa43b8b9c74725889e97fc4a8e26d1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2caa43b8b9c74725889e97fc4a8e26d1" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 38737 } } peers { permanent_uuid: "fe28fc3af9c24c29b272bb8c54f875ea" member_type: VOTER last_known_addr { host: "127.31.100.61" port: 42987 } } peers { permanent_uuid: "6e028db49a0243dd9baf3c86319ab253" member_type: VOTER last_known_addr { host: "127.31.100.60" port: 44447 } } }
I20260812 05:24:50.907485 5996 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6e028db49a0243dd9baf3c86319ab253 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 05:24:50.907873 6054 mvcc.cc:205] Tried to move back new op lower bound from 7317554343509778432 to 7317554343487516672. Current Snapshot: MvccSnapshot[applied={T|T < 7317554343509778432}]
I20260812 05:24:50.908684 6061 mvcc.cc:205] Tried to move back new op lower bound from 7317554343509778432 to 7317554343487516672. Current Snapshot: MvccSnapshot[applied={T|T < 7317554343509778432}]
I20260812 05:24:50.909137 6049 catalog_manager.cc:1347] Generated new cluster ID: 0cd48f20e0084d82b1cd3ecd64ed9736
I20260812 05:24:50.909193 6049 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 05:24:50.911057 5993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:24:50.892941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49904 (local address 127.31.100.60:44447)
0812 05:24:50.892979 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0812 05:24:50.892980 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0812 05:24:50.896332 (+ 3352us) server_negotiation.cc:407] Connection header received
0812 05:24:50.896421 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:24:50.896422 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:24:50.896431 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:24:50.896458 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:24:50.899908 (+ 3450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:24:50.900617 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:24:50.902720 (+ 2103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:24:50.903014 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:24:50.903052 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:24:50.907399 (+ 4347us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 05:24:50.907401 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 05:24:50.907401 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 05:24:50.907408 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 05:24:50.909223 (+ 1815us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:24:50.909872 (+ 649us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:24:50.909874 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:24:50.909874 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:24:50.909930 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:24:50.910226 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:24:50.910227 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:24:50.910227 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:24:50.910477 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 05:24:50.910573 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0812 05:24:50.910895 (+ 322us) server_negotiation.cc:299] Negotiation successful
0812 05:24:50.910976 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260812 05:24:50.912940 5866 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.100.60" port: 44447 } attempt: 3
Aug 12 05:24:50 dist-test-slave-42z9 krb5kdc[5602](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786512290, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.100.61@KRBTEST.COM for kudu/127.31.100.60@KRBTEST.COM
I20260812 05:24:50.921429 5873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:24:50.913164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.100.60:44447 (local address 127.0.0.1:49912)
0812 05:24:50.913346 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0812 05:24:50.913355 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0812 05:24:50.913398 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 05:24:50.913692 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 05:24:50.913693 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 05:24:50.913841 (+ 148us) client_negotiation.cc:189] Negotiated authn=SASL
0812 05:24:50.913881 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:24:50.913883 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:24:50.914787 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:24:50.914788 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:24:50.915595 (+ 807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 05:24:50.915598 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 05:24:50.916116 (+ 518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 05:24:50.916118 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 05:24:50.916192 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:24:50.916645 (+ 453us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 05:24:50.916650 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0812 05:24:50.918127 (+ 1477us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 05:24:50.920061 (+ 1934us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 05:24:50.920062 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 05:24:50.920063 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 05:24:50.920282 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 05:24:50.920572 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 05:24:50.920573 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 05:24:50.920573 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 05:24:50.920642 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 05:24:50.921105 (+ 463us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 05:24:50.921106 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 05:24:50.921165 (+ 59us) client_negotiation.cc:769] Sending connection context
0812 05:24:50.921225 (+ 60us) client_negotiation.cc:240] Negotiation successful
0812 05:24:50.921338 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166}
I20260812 05:24:50.922268 5993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:24:50.913223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49912 (local address 127.31.100.60:44447)
0812 05:24:50.913272 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0812 05:24:50.913274 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 05:24:50.913469 (+ 195us) server_negotiation.cc:407] Connection header received
0812 05:24:50.913488 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:24:50.913489 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:24:50.913507 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:24:50.913551 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:24:50.913958 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:24:50.914682 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:24:50.915699 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:24:50.916000 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:24:50.916054 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:24:50.918231 (+ 2177us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 05:24:50.918232 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 05:24:50.918232 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 05:24:50.918238 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0812 05:24:50.919959 (+ 1721us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:24:50.920370 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:24:50.920371 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:24:50.920371 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:24:50.920492 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 05:24:50.920720 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 05:24:50.920721 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 05:24:50.920721 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 05:24:50.920974 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 05:24:50.921892 (+ 918us) server_negotiation.cc:1035] Waiting for connection context
0812 05:24:50.921966 (+ 74us) server_negotiation.cc:299] Negotiation successful
0812 05:24:50.922099 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260812 05:24:50.923583 6049 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 05:24:50.923939 5866 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fe28fc3af9c24c29b272bb8c54f875ea: Bootstrap starting.
I20260812 05:24:50.924233 6049 catalog_manager.cc:1504] Loading token signing keys...
I20260812 05:24:50.924626 5866 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fe28fc3af9c24c29b272bb8c54f875ea: Neither blocks nor log segments found. Creating new log.
I20260812 05:24:50.924908 5866 log.cc:824] T 00000000000000000000000000000000 P fe28fc3af9c24c29b272bb8c54f875ea: Log is configured to *not* fsync() on all Append() calls
I20260812 05:24:50.929221 6049 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2caa43b8b9c74725889e97fc4a8e26d1: Generated new TSK 0
I20260812 05:24:50.929381 6049 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 05:24:50.930088 5866 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fe28fc3af9c24c29b272bb8c54f875ea: No bootstrap required, opened a new log
I20260812 05:24:50.932150 5866 raft_consensus.cc:348] T 00000000000000000000000000000000 P fe28fc3af9c24c29b272bb8c54f875ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2caa43b8b9c74725889e97fc4a8e26d1" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 38737 } } peers { permanent_uuid: "fe28fc3af9c24c29b272bb8c54f875ea" member_type: VOTER last_known_addr { host: "127.31.100.61" port: 42987 } } peers { permanent_uuid: "6e028db49a0243dd9baf3c86319ab253" member_type: VOTER last_known_addr { host: "127.31.100.60" port: 44447 } }
I20260812 05:24:50.932296 5866 raft_consensus.cc:729] T 00000000000000000000000000000000 P fe28fc3af9c24c29b272bb8c54f875ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe28fc3af9c24c29b272bb8c54f875ea, State: Initialized, Role: FOLLOWER
I20260812 05:24:50.932386 5866 consensus_queue.cc:260] T 00000000000000000000000000000000 P fe28fc3af9c24c29b272bb8c54f875ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2caa43b8b9c74725889e97fc4a8e26d1" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 38737 } } peers { permanent_uuid: "fe28fc3af9c24c29b272bb8c54f875ea" member_type: VOTER last_known_addr { host: "127.31.100.61" port: 42987 } } peers { permanent_uuid: "6e028db49a0243dd9baf3c86319ab253" member_type: VOTER last_known_addr { host: "127.31.100.60" port: 44447 } }
I20260812 05:24:50.932745 5866 sys_catalog.cc:564] T 00000000000000000000000000000000 P fe28fc3af9c24c29b272bb8c54f875ea [sys.catalog]: configured and running, proceeding with master startup.
I20260812 05:24:50.932811 5826 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2caa43b8b9c74725889e97fc4a8e26d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "fe28fc3af9c24c29b272bb8c54f875ea"
I20260812 05:24:50.932931 5826 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fe28fc3af9c24c29b272bb8c54f875ea [term 0 FOLLOWER]: Advancing to term 1
I20260812 05:24:50.933773 5826 raft_consensus.cc:2454] T 00000000000000000000000000000000 P fe28fc3af9c24c29b272bb8c54f875ea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2caa43b8b9c74725889e97fc4a8e26d1 in term 1.
I20260812 05:24:50.934125 5824 raft_consensus.cc:1261] T 00000000000000000000000000000000 P fe28fc3af9c24c29b272bb8c54f875ea [term 1 FOLLOWER]: Refusing update from remote peer 2caa43b8b9c74725889e97fc4a8e26d1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 05:24:50.934171 6077 sys_catalog.cc:455] T 00000000000000000000000000000000 P fe28fc3af9c24c29b272bb8c54f875ea [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2caa43b8b9c74725889e97fc4a8e26d1" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 38737 } } peers { permanent_uuid: "fe28fc3af9c24c29b272bb8c54f875ea" member_type: VOTER last_known_addr { host: "127.31.100.61" port: 42987 } } peers { permanent_uuid: "6e028db49a0243dd9baf3c86319ab253" member_type: VOTER last_known_addr { host: "127.31.100.60" port: 44447 } } }
I20260812 05:24:50.934592 6017 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 2caa43b8b9c74725889e97fc4a8e26d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe28fc3af9c24c29b272bb8c54f875ea" member_type: VOTER last_known_addr { host: "127.31.100.61" port: 42987 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 05:24:50.934646 6077 sys_catalog.cc:458] T 00000000000000000000000000000000 P fe28fc3af9c24c29b272bb8c54f875ea [sys.catalog]: This master's current role is: FOLLOWER
W20260812 05:24:50.935335 6092 catalog_manager.cc:1558] T 00000000000000000000000000000000 P fe28fc3af9c24c29b272bb8c54f875ea: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 05:24:50.935387 6092 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 05:24:50.936468 6077 sys_catalog.cc:455] T 00000000000000000000000000000000 P fe28fc3af9c24c29b272bb8c54f875ea [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2caa43b8b9c74725889e97fc4a8e26d1. Latest consensus state: current_term: 1 leader_uuid: "2caa43b8b9c74725889e97fc4a8e26d1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2caa43b8b9c74725889e97fc4a8e26d1" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 38737 } } peers { permanent_uuid: "fe28fc3af9c24c29b272bb8c54f875ea" member_type: VOTER last_known_addr { host: "127.31.100.61" port: 42987 } } peers { permanent_uuid: "6e028db49a0243dd9baf3c86319ab253" member_type: VOTER last_known_addr { host: "127.31.100.60" port: 44447 } } }
I20260812 05:24:50.936599 6077 sys_catalog.cc:458] T 00000000000000000000000000000000 P fe28fc3af9c24c29b272bb8c54f875ea [sys.catalog]: This master's current role is: FOLLOWER
I20260812 05:24:50.936672 6077 sys_catalog.cc:455] T 00000000000000000000000000000000 P fe28fc3af9c24c29b272bb8c54f875ea [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2caa43b8b9c74725889e97fc4a8e26d1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2caa43b8b9c74725889e97fc4a8e26d1" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 38737 } } peers { permanent_uuid: "fe28fc3af9c24c29b272bb8c54f875ea" member_type: VOTER last_known_addr { host: "127.31.100.61" port: 42987 } } peers { permanent_uuid: "6e028db49a0243dd9baf3c86319ab253" member_type: VOTER last_known_addr { host: "127.31.100.60" port: 44447 } } }
I20260812 05:24:50.936800 6077 sys_catalog.cc:458] T 00000000000000000000000000000000 P fe28fc3af9c24c29b272bb8c54f875ea [sys.catalog]: This master's current role is: FOLLOWER
I20260812 05:24:50.938208 6094 mvcc.cc:205] Tried to move back new op lower bound from 7317554343639519232 to 7317554343487516672. Current Snapshot: MvccSnapshot[applied={T|T < 7317554343509778432}]
W20260812 05:24:50.967331 5749 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 05:24:50.965798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46542 (local address 127.31.100.62:38737)
0812 05:24:50.965842 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0812 05:24:50.965844 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 05:24:50.966672 (+ 828us) server_negotiation.cc:407] Connection header received
0812 05:24:50.966771 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:24:50.966772 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:24:50.966783 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:24:50.966812 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:24:50.967265 (+ 453us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46542: BlockingRecv error: recv got EOF from 127.0.0.1:46542 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260812 05:24:50.968168 5875 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 05:24:50.966159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50908 (local address 127.31.100.61:42987)
0812 05:24:50.966197 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0812 05:24:50.966199 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 05:24:50.967692 (+ 1493us) server_negotiation.cc:407] Connection header received
0812 05:24:50.967786 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:24:50.967787 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:24:50.967796 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:24:50.967822 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:24:50.968110 (+ 288us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50908: BlockingRecv error: recv got EOF from 127.0.0.1:50908 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260812 05:24:50.968928 5993 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 05:24:50.966427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49922 (local address 127.31.100.60:44447)
0812 05:24:50.966466 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0812 05:24:50.966468 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 05:24:50.968474 (+ 2006us) server_negotiation.cc:407] Connection header received
0812 05:24:50.968559 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:24:50.968560 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:24:50.968571 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:24:50.968597 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:24:50.968873 (+ 276us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49922: BlockingRecv error: recv got EOF from 127.0.0.1:49922 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260812 05:24:50.971204 5875 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 05:24:50.970183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50920 (local address 127.31.100.61:42987)
0812 05:24:50.970213 (+ 30us) server_negotiation.cc:206] Beginning negotiation
0812 05:24:50.970215 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 05:24:50.970694 (+ 479us) server_negotiation.cc:407] Connection header received
0812 05:24:50.970814 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:24:50.970815 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:24:50.970833 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:24:50.970871 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:24:50.971164 (+ 293us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50920: BlockingRecv error: recv got EOF from 127.0.0.1:50920 (error 108)
Metrics: {"server-negotiator.queue_time_us":14}
W20260812 05:24:50.974174 5875 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 05:24:50.973482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50934 (local address 127.31.100.61:42987)
0812 05:24:50.973515 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0812 05:24:50.973516 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0812 05:24:50.973707 (+ 191us) server_negotiation.cc:407] Connection header received
0812 05:24:50.973794 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:24:50.973795 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:24:50.973805 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:24:50.973831 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:24:50.974136 (+ 305us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50934: BlockingRecv error: recv got EOF from 127.0.0.1:50934 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260812 05:24:50.978155 5875 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 05:24:50.977329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50950 (local address 127.31.100.61:42987)
0812 05:24:50.977361 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0812 05:24:50.977362 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0812 05:24:50.977671 (+ 309us) server_negotiation.cc:407] Connection header received
0812 05:24:50.977767 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:24:50.977768 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:24:50.977777 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:24:50.977803 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:24:50.978118 (+ 315us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50950: BlockingRecv error: recv got EOF from 127.0.0.1:50950 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260812 05:24:50.984089 5875 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 05:24:50.983360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50966 (local address 127.31.100.61:42987)
0812 05:24:50.983392 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0812 05:24:50.983394 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 05:24:50.983606 (+ 212us) server_negotiation.cc:407] Connection header received
0812 05:24:50.983692 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:24:50.983693 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:24:50.983703 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:24:50.983729 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:24:50.984048 (+ 319us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50966: BlockingRecv error: recv got EOF from 127.0.0.1:50966 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260812 05:24:50.994192 5875 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 05:24:50.993337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50982 (local address 127.31.100.61:42987)
0812 05:24:50.993373 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0812 05:24:50.993374 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0812 05:24:50.993639 (+ 265us) server_negotiation.cc:407] Connection header received
0812 05:24:50.993734 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:24:50.993735 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:24:50.993746 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:24:50.993774 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:24:50.994145 (+ 371us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50982: BlockingRecv error: recv got EOF from 127.0.0.1:50982 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260812 05:24:51.012492 5875 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 05:24:51.011617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50984 (local address 127.31.100.61:42987)
0812 05:24:51.011659 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0812 05:24:51.011661 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 05:24:51.011923 (+ 262us) server_negotiation.cc:407] Connection header received
0812 05:24:51.012026 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:24:51.012027 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:24:51.012038 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:24:51.012067 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:24:51.012435 (+ 368us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50984: BlockingRecv error: recv got EOF from 127.0.0.1:50984 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260812 05:24:51.046909 5875 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 05:24:51.046199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50994 (local address 127.31.100.61:42987)
0812 05:24:51.046248 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0812 05:24:51.046250 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 05:24:51.046360 (+ 110us) server_negotiation.cc:407] Connection header received
0812 05:24:51.046433 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:24:51.046434 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:24:51.046445 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:24:51.046489 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:24:51.046826 (+ 337us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50994: BlockingRecv error: recv got EOF from 127.0.0.1:50994 (error 108)
Metrics: {"server-negotiator.queue_time_us":25}
W20260812 05:24:51.112664 5875 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 05:24:51.111986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51000 (local address 127.31.100.61:42987)
0812 05:24:51.112035 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0812 05:24:51.112038 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 05:24:51.112139 (+ 101us) server_negotiation.cc:407] Connection header received
0812 05:24:51.112214 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:24:51.112215 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:24:51.112226 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:24:51.112273 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:24:51.112599 (+ 326us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51000: BlockingRecv error: recv got EOF from 127.0.0.1:51000 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
W20260812 05:24:51.242458 5875 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 05:24:51.241768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51014 (local address 127.31.100.61:42987)
0812 05:24:51.241832 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0812 05:24:51.241835 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 05:24:51.241918 (+ 83us) server_negotiation.cc:407] Connection header received
0812 05:24:51.241997 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:24:51.241998 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:24:51.242010 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:24:51.242053 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:24:51.242392 (+ 339us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51014: BlockingRecv error: recv got EOF from 127.0.0.1:51014 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
W20260812 05:24:51.500276 5875 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 05:24:51.499615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51020 (local address 127.31.100.61:42987)
0812 05:24:51.499675 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0812 05:24:51.499677 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 05:24:51.499713 (+ 36us) server_negotiation.cc:407] Connection header received
0812 05:24:51.499795 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:24:51.499796 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:24:51.499808 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:24:51.499854 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0812 05:24:51.500208 (+ 354us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51020: BlockingRecv error: recv got EOF from 127.0.0.1:51020 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
I20260812 05:24:51.523952 6214 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2caa43b8b9c74725889e97fc4a8e26d1 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2caa43b8b9c74725889e97fc4a8e26d1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2caa43b8b9c74725889e97fc4a8e26d1" member_type: VOTER last_known_addr { host: "127.31.100.62" port: 38737 } } peers { permanent_uuid: "fe28fc3af9c24c29b272bb8c54f875ea" member_type: VOTER last_known_addr { host: "127.31.100.61" port: 42987 } } peers { permanent_uuid: "6e028db49a0243dd9baf3c86319ab253" member_type: VOTER last_known_addr { host: "127.31.100.60" port: 44447 } } }
I20260812 05:24:51.524073 6214 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2caa43b8b9c74725889e97fc4a8e26d1 [sys.catalog]: This master's current role is: LEADER
I20260812 05:24:51.856215 6013 catalog_manager.cc:1259] Loaded cluster ID: 0cd48f20e0084d82b1cd3ecd64ed9736
I20260812 05:24:51.856287 6013 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 6e028db49a0243dd9baf3c86319ab253: loading cluster ID for follower catalog manager: success
I20260812 05:24:51.857084 6013 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 6e028db49a0243dd9baf3c86319ab253: acquiring CA information for follower catalog manager: success
I20260812 05:24:51.857388 6013 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 6e028db49a0243dd9baf3c86319ab253: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260812 05:24:51.935789 6092 catalog_manager.cc:1259] Loaded cluster ID: 0cd48f20e0084d82b1cd3ecd64ed9736
I20260812 05:24:51.935868 6092 catalog_manager.cc:1552] T 00000000000000000000000000000000 P fe28fc3af9c24c29b272bb8c54f875ea: loading cluster ID for follower catalog manager: success
I20260812 05:24:51.936645 6092 catalog_manager.cc:1574] T 00000000000000000000000000000000 P fe28fc3af9c24c29b272bb8c54f875ea: acquiring CA information for follower catalog manager: success
I20260812 05:24:51.936936 6092 catalog_manager.cc:1602] T 00000000000000000000000000000000 P fe28fc3af9c24c29b272bb8c54f875ea: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260812 05:24:52.015569 6226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:24:52.013289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51024 (local address 127.31.100.61:42987)
0812 05:24:52.013391 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0812 05:24:52.013393 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 05:24:52.013420 (+ 27us) server_negotiation.cc:407] Connection header received
0812 05:24:52.013501 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:24:52.013501 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:24:52.013553 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:24:52.013646 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 05:24:52.013769 (+ 123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:24:52.014262 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:24:52.014959 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:24:52.015141 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:24:52.015183 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:24:52.015296 (+ 113us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 05:24:52.015404 (+ 108us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 05:24:52.015456 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0812 05:24:52.015471 (+ 15us) server_negotiation.cc:299] Negotiation successful
0812 05:24:52.015486 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85,"thread_start_us":57,"threads_started":1}
I20260812 05:24:52.019421 6232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:24:52.017251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49938 (local address 127.31.100.60:44447)
0812 05:24:52.017358 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0812 05:24:52.017360 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 05:24:52.017367 (+ 7us) server_negotiation.cc:407] Connection header received
0812 05:24:52.017403 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:24:52.017404 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:24:52.017461 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:24:52.017526 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 05:24:52.017604 (+ 78us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:24:52.018076 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:24:52.018780 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:24:52.019009 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:24:52.019058 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:24:52.019175 (+ 117us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 05:24:52.019266 (+ 91us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 05:24:52.019316 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0812 05:24:52.019333 (+ 17us) server_negotiation.cc:299] Negotiation successful
0812 05:24:52.019350 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87,"thread_start_us":54,"threads_started":1}
I20260812 05:24:52.023458 6226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:24:52.020990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51038 (local address 127.31.100.61:42987)
0812 05:24:52.021035 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0812 05:24:52.021037 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 05:24:52.021121 (+ 84us) server_negotiation.cc:407] Connection header received
0812 05:24:52.021220 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:24:52.021221 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:24:52.021231 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:24:52.021258 (+ 27us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 05:24:52.021407 (+ 149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:24:52.021910 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:24:52.022796 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:24:52.023069 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:24:52.023133 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:24:52.023245 (+ 112us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 05:24:52.023275 (+ 30us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 05:24:52.023320 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0812 05:24:52.023366 (+ 46us) server_negotiation.cc:299] Negotiation successful
0812 05:24:52.023386 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260812 05:24:52.023893 6232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:24:52.021226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49944 (local address 127.31.100.60:44447)
0812 05:24:52.021263 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0812 05:24:52.021265 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 05:24:52.021271 (+ 6us) server_negotiation.cc:407] Connection header received
0812 05:24:52.021290 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:24:52.021291 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:24:52.021300 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:24:52.021341 (+ 41us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 05:24:52.021503 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:24:52.022195 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:24:52.023050 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:24:52.023313 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:24:52.023350 (+ 37us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:24:52.023615 (+ 265us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 05:24:52.023646 (+ 31us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 05:24:52.023694 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0812 05:24:52.023802 (+ 108us) server_negotiation.cc:299] Negotiation successful
0812 05:24:52.023826 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260812 05:24:52.024029 6239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 05:24:52.020882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46550 (local address 127.31.100.62:38737)
0812 05:24:52.021012 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0812 05:24:52.021014 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 05:24:52.021024 (+ 10us) server_negotiation.cc:407] Connection header received
0812 05:24:52.021069 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 05:24:52.021070 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 05:24:52.021123 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 05:24:52.021333 (+ 210us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 05:24:52.021372 (+ 39us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:24:52.022168 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:24:52.023095 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 05:24:52.023387 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 05:24:52.023596 (+ 209us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 05:24:52.023671 (+ 75us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 05:24:52.023772 (+ 101us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 05:24:52.023860 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0812 05:24:52.023902 (+ 42us) server_negotiation.cc:299] Negotiation successful
0812 05:24:52.023927 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107,"thread_start_us":51,"threads_started":1}
I20260812 05:24:52.025594 32144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0waFvT/build/release/bin/kudu with pid 5623
I20260812 05:24:52.045214 32144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0waFvT/build/release/bin/kudu with pid 5756
I20260812 05:24:52.055794 32144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0waFvT/build/release/bin/kudu with pid 5879
2026-08-12T05:24:52Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (2940 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (5922 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (274455 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST